Oct 30 21:08:39.690736 ip-10-0-133-149 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:08:40.144499 ip-10-0-133-149 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:08:40.144499 ip-10-0-133-149 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:08:40.144499 ip-10-0-133-149 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:08:40.144499 ip-10-0-133-149 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:08:40.150605 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150525 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.150605 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150603 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.150605 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150609 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150612 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150616 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150619 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150622 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150625 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150627 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150630 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150633 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150635 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150638 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150640 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150644 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150646 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150649 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150652 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150654 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150656 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150659 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.150698 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150662 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150665 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150667 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150670 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150672 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150675 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150677 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150680 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150682 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150685 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150687 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150689 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150692 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150694 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150698 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150701 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150704 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150706 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150709 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.151131 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150712 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150715 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150718 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150720 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150722 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150725 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150727 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150730 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150732 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150735 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150738 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150741 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150743 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150746 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150749 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150751 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150754 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150756 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150759 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.151586 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150761 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150764 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150768 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150772 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150775 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150778 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150781 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150784 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150788 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150790 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150793 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150795 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150798 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150801 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150804 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150806 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150808 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150811 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150814 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150816 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.152019 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150819 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150821 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150823 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150826 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150829 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150831 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150834 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150838 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150842 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150844 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150847 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150849 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150852 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150854 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150857 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150861 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150863 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150865 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150868 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150871 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.152489 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150875 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150878 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150880 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150883 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150885 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150888 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.150891 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151638 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151645 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151649 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151654 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151657 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151659 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151662 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151665 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151668 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151672 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151676 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151679 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.152950 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151682 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151684 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151687 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151690 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151693 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151696 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151699 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151701 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151704 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151707 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151709 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151712 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151715 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151718 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151720 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151723 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151725 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151727 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151730 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.153410 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151732 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151735 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151738 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151740 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151743 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151745 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151748 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151750 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151752 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151755 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151757 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151760 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151762 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151765 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151767 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151770 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151773 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151775 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151778 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151781 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.153861 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151783 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151786 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151788 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151791 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151796 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151798 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151801 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151803 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151806 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151809 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151811 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151814 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151816 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151818 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151822 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151825 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151827 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151830 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151832 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151835 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.154318 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151837 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151840 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151842 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151845 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151848 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151850 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151853 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151855 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151857 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151860 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151863 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151865 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151868 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151870 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151873 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151875 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151879 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151882 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151884 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151887 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.154804 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151889 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151892 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151894 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151897 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151899 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151902 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151904 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151907 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151910 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151912 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151915 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151918 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151920 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151923 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.151925 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152416 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152422 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152424 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152427 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.155267 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152429 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152432 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152435 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152439 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152441 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152444 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152447 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152449 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152452 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152455 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152458 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152460 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152462 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152465 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152467 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152470 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152472 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152475 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152477 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152480 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.155734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152483 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152485 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152488 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152490 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152493 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152496 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152498 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152501 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152503 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152506 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152508 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152511 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152513 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152515 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152520 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152524 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152527 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152530 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152533 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.156205 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152537 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152540 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152542 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152545 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152548 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152550 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152553 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152555 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152558 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152561 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152563 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152566 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152568 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152571 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152574 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152576 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152579 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152581 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152584 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152587 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.156673 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152589 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152592 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152595 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152597 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152600 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152602 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152605 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152607 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152609 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152612 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152614 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152617 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152619 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152622 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152624 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152627 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152629 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152632 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152634 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152637 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.157138 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152639 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152642 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152644 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152647 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152650 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152654 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152657 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152661 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152663 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152666 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152669 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152672 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152675 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152678 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152680 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152682 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152685 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152687 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152690 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.157694 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152692 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152694 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152697 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.152699 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153745 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153754 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153761 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153766 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153770 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153774 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153778 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153782 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153785 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153787 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153791 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153794 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153797 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153800 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153802 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153805 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153808 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153812 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153816 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:08:40.158169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153818 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153822 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153825 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153828 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153832 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153835 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153838 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153841 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153844 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153847 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153850 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153853 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153856 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153862 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153865 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153868 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153871 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153875 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153877 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153882 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153885 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153888 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153891 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153894 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:08:40.158753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153898 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153901 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153904 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153907 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153910 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153913 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153916 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153919 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153922 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153925 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153927 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153932 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153934 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153938 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153941 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153943 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153947 2576 flags.go:64] FLAG: --help="false" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153950 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153953 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153956 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153959 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153963 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153966 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153969 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:08:40.159368 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153972 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153975 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153978 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153980 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153983 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153986 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153989 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153992 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153995 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.153997 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154000 2576 flags.go:64] FLAG: --lock-file="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154003 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154006 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154009 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154017 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154020 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154025 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154028 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154031 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154034 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154037 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154040 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154045 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154047 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154051 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 21:08:40.159947 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154054 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154057 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154060 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154063 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154067 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154070 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154073 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154080 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154083 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154086 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154089 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154091 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154094 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154097 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154100 2576 flags.go:64] FLAG: --port="10250" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154103 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154106 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-041c6ccd130168185" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154109 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154112 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154114 2576 flags.go:64] FLAG: --register-node="true" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154117 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154121 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154124 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154128 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154131 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154135 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:08:40.160591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154138 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154141 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154144 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154147 2576 flags.go:64] FLAG: --runonce="false" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154149 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154152 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154155 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154158 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154161 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154164 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154169 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154172 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154176 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154178 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154181 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154184 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154187 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154190 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154193 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154198 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154201 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154203 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154207 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154210 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154213 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:08:40.161179 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154215 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154218 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154221 2576 flags.go:64] FLAG: --v="2" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154226 2576 flags.go:64] FLAG: --version="false" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154231 2576 flags.go:64] FLAG: --vmodule="" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154235 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154239 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154373 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154377 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154381 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154383 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154396 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154400 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154402 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154405 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154408 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154410 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154415 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154417 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154420 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154423 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.161763 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154425 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154427 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154430 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154448 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154452 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154455 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154458 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154461 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154464 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154466 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154469 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154471 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154474 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154476 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154479 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154482 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154485 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154487 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154490 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.162251 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154493 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154496 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154498 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154501 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154503 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154506 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154509 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154511 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154519 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154524 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154527 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154529 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154532 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154534 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154537 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154539 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154542 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154545 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154549 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.163033 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154552 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154555 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154557 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154560 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154562 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154565 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154568 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154570 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154573 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154576 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154579 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154581 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154585 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154589 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154592 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154595 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154597 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154600 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154602 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.163779 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154605 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154607 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154610 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154614 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154616 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154619 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154621 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154624 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154626 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154629 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154631 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154634 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154636 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154639 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154641 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154644 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154646 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154649 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154651 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154654 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.164214 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154656 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154659 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154666 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154669 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154671 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154674 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154676 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154679 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154682 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154685 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154688 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154690 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154693 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154695 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154698 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.154726 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:08:40.164687 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154761 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154765 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154768 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154771 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154774 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154776 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154779 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154782 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154784 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154787 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154789 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154792 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154794 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154797 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154799 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154802 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154805 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154807 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154812 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154814 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.165059 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154817 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154820 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154822 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154825 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154828 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154830 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154833 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154836 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154838 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154841 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154844 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154846 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154849 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154852 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154854 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154857 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154859 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154861 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154864 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.165540 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154866 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154869 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154872 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154875 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154877 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154879 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154882 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154885 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154887 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154890 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154892 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154896 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154898 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154901 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154903 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154905 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154908 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154910 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154913 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.165988 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154916 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154920 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154923 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154926 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154929 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154932 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154934 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154937 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154940 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154942 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154944 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154947 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154949 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154951 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154956 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154959 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154961 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154964 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154966 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.166445 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154969 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154971 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154973 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154976 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154978 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154982 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154984 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154987 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154989 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154991 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154994 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154996 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.154999 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155002 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155004 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155007 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155009 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155011 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155014 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155016 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.166894 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155019 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155021 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155024 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155026 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155028 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155031 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155033 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155035 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.155038 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.163674 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.163690 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.163707 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.163712 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163773 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163778 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163781 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163784 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163787 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163790 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.167333 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163793 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163795 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163798 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163800 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163803 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163805 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163808 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163810 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163813 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163815 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163818 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163821 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163823 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163826 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163828 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163836 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163838 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163841 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163844 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163847 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.167790 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163849 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163852 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163854 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163857 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163860 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163864 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163867 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163870 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163874 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163876 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163879 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163881 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163884 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163887 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163889 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163892 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163895 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163897 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163900 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163902 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.168239 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163905 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163908 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163910 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163913 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163915 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163918 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163921 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163924 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163926 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163929 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163931 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163934 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163936 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163939 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163941 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163944 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163947 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163949 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163952 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163954 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.168787 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163957 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163959 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163961 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163964 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163966 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163969 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163971 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163973 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163976 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163978 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163981 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163983 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163986 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163989 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163992 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163994 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163997 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.163999 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164002 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.169247 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164005 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164007 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164010 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164012 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164015 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164017 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164020 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164023 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164025 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164028 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164030 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164033 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164036 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164038 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164041 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164043 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164046 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164049 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164052 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.169734 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164056 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164058 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.164086 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164227 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164232 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164236 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164239 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164242 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164245 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164247 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164250 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164253 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164255 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164258 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164261 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164266 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.170182 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164269 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164271 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164273 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164276 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164279 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164281 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164284 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164286 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164288 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164291 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164293 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164296 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164298 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164301 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164303 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164306 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164308 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164311 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164313 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164315 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.170578 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164318 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164321 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164323 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164326 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164328 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164331 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164334 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164336 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164339 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164342 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164344 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164346 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164349 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164351 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164354 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164356 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164359 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164361 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164363 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164366 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.171042 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164368 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164370 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164373 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164376 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164378 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164380 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164383 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164385 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164400 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164403 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164405 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164407 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164410 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164413 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164415 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164419 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164422 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164425 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.171506 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164428 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164431 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164433 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164436 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164439 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164441 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164444 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164447 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164449 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164451 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164454 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164456 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164459 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164461 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164464 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164467 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164469 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164472 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164474 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164477 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.171960 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164479 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164482 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164484 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164486 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164489 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164491 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164494 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164496 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164498 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164501 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164503 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164505 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164508 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164511 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164514 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.164537 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:08:40.172550 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164566 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164570 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164574 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164577 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164580 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164582 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164585 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164587 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164590 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164592 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164595 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164597 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164599 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164603 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164607 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164610 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164613 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164615 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164618 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164620 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:08:40.172930 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164623 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164625 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164627 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164630 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164632 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164634 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164637 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164639 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164642 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164644 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164648 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164650 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164653 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164655 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164658 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164660 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164662 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164665 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164667 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164669 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:08:40.173437 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164672 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164675 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164677 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164679 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164682 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164684 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164686 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164690 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164692 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164695 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164697 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164700 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164703 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164705 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164707 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164710 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164712 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164714 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164717 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:08:40.173900 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164720 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164722 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164725 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164728 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164731 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164734 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164736 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164738 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164741 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164743 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164746 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164750 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164753 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164756 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164758 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164761 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164763 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164766 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164768 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:08:40.174314 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164772 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164774 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164777 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164779 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164782 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164784 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164789 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164792 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164794 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164797 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164799 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164802 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164804 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164807 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164809 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164811 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164814 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164816 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164819 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:08:40.174780 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164822 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164824 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164827 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164829 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164832 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164834 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164837 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164839 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:40.164841 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.164895 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.169546 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.170660 2576 server.go:1014] "Starting client certificate rotation" Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.170738 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.170770 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:08:40.175241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.172934 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:08:40.177047 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.177031 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:08:40.186272 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.186254 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:08:40.190226 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.190213 2576 log.go:25] "Validated CRI v1 image API" Oct 30 21:08:40.191081 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.191056 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:08:40.195024 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.194992 2576 fs.go:135] Filesystem UUIDs: map[6289a759-81b1-426e-bec7-40134eecedc5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bc089313-3d17-4db7-bc12-f8c2c2c6a83f:/dev/nvme0n1p4] Oct 30 21:08:40.195077 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.195027 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:08:40.201354 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.201228 2576 manager.go:217] Machine: {Timestamp:2026-10-30 21:08:40.199263179 +0000 UTC m=+0.380183068 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200384 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c1040ef495967deb6b62a4a9811f SystemUUID:ec24c104-0ef4-9596-7deb-6b62a4a9811f BootID:d17a07ac-3230-469a-9b8f-d115bb44b42c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e3:93:51:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e3:93:51:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:10:c5:3e:bd:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:08:40.201354 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.201342 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:08:40.201497 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.201467 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:08:40.202681 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.202651 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:08:40.202812 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.202681 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:08:40.202857 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.202814 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:08:40.202857 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.202822 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:08:40.202906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.202856 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:08:40.202906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.202870 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:08:40.204180 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.204170 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:08:40.208896 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.208884 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:08:40.209010 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.209001 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:08:40.209870 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.209859 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:08:40.209906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.209877 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:08:40.209906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.209891 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:08:40.209906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.209902 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:08:40.209991 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.209913 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:08:40.210298 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.210287 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:08:40.212739 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.212722 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:08:40.219829 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.219817 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:08:40.220807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220794 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:08:40.220841 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220826 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:08:40.220971 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220962 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:08:40.221005 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220976 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:08:40.221005 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220982 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:08:40.221005 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220987 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:08:40.221005 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220992 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:08:40.221005 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.220998 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:08:40.221687 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.221678 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:08:40.221721 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.221688 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:08:40.221721 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.221696 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:08:40.221721 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.221703 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:08:40.221721 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.221717 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:08:40.221836 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.221726 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:08:40.223049 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.223040 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:08:40.223084 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.223050 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:08:40.226909 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.226895 2576 server.go:1263] "Started kubelet" Oct 30 21:08:40.227020 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.226967 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:08:40.227084 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.226994 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:08:40.227439 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.227418 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:08:40.227749 ip-10-0-133-149 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:08:40.229655 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.229641 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:08:40.230431 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.230412 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:08:40.234926 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.234906 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:08:40.237488 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.237472 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:08:40.238759 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.238745 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:08:40.238922 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.238899 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.239526 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239510 2576 factory.go:55] Registering systemd factory Oct 30 21:08:40.239526 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239526 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:08:40.239657 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239572 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:08:40.239657 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239572 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:08:40.239657 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239594 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:08:40.239753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239688 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:08:40.239753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239695 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:08:40.239753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239745 2576 factory.go:153] Registering CRI-O factory Oct 30 21:08:40.239824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239755 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 21:08:40.239824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239807 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:08:40.239824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239821 2576 factory.go:103] Registering Raw factory Oct 30 21:08:40.239906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.239846 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 21:08:40.240961 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.240947 2576 manager.go:319] Starting recovery of all containers Oct 30 21:08:40.253353 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.253335 2576 manager.go:324] Recovery completed Oct 30 21:08:40.256280 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.256260 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:08:40.257583 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.257563 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:08:40.257583 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.257567 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:08:40.258369 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.258357 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.258705 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.257565 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18e36ad07841366b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-10-30 21:08:40.226870891 +0000 UTC m=+0.407790779,LastTimestamp:2026-10-30 21:08:40.226870891 +0000 UTC m=+0.407790779,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Oct 30 21:08:40.259918 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.259901 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.259972 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.259939 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.259972 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.259953 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.260538 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.260521 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:08:40.260538 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.260532 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.260667 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.260533 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:08:40.260667 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.260609 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:08:40.261169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.261155 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.261225 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.261181 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.261225 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.261194 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.274954 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.274935 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:08:40.275041 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.274980 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18e36ad07a39824a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-10-30 21:08:40.259920458 +0000 UTC m=+0.440840349,LastTimestamp:2026-10-30 21:08:40.259920458 +0000 UTC m=+0.440840349,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Oct 30 21:08:40.275114 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.275094 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:08:40.275231 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.275213 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:08:40.276673 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.276654 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2wpgk" Oct 30 21:08:40.282465 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.282020 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2wpgk" Oct 30 21:08:40.300032 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.299969 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18e36ad07a39e8c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-10-30 21:08:40.259946694 +0000 UTC m=+0.440866581,LastTimestamp:2026-10-30 21:08:40.259946694 +0000 UTC m=+0.440866581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Oct 30 21:08:40.312823 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.312762 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18e36ad07a3a111e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-10-30 21:08:40.259957022 +0000 UTC m=+0.440876909,LastTimestamp:2026-10-30 21:08:40.259957022 +0000 UTC m=+0.440876909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Oct 30 21:08:40.339137 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.339114 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.349759 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.349692 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-149.ec2.internal.18e36ad07a39824a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18e36ad07a39824a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-10-30 21:08:40.259920458 +0000 UTC m=+0.440840349,LastTimestamp:2026-10-30 21:08:40.261168872 +0000 UTC m=+0.442088759,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Oct 30 21:08:40.394286 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.394213 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-149.ec2.internal.18e36ad07a39e8c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-149.ec2.internal.18e36ad07a39e8c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-149.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-149.ec2.internal,},FirstTimestamp:2026-10-30 21:08:40.259946694 +0000 UTC m=+0.440866581,LastTimestamp:2026-10-30 21:08:40.261187512 +0000 UTC m=+0.442107403,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-149.ec2.internal,}" Oct 30 21:08:40.396339 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.396291 2576 policy_none.go:50] "Start" Oct 30 21:08:40.396339 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.396312 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:08:40.396339 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.396324 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:08:40.398281 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.398264 2576 policy_none.go:44] "Start" Oct 30 21:08:40.421568 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.418056 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:08:40.421568 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.419493 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:08:40.421568 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.419514 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:08:40.421568 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.419535 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:08:40.421568 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.419574 2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:08:40.439310 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.439293 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.440445 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.440418 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:08:40.441442 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.441429 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:08:40.442988 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.442976 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 21:08:40.443061 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.443007 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:08:40.443061 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.443018 2576 server.go:87] "Starting device plugin registration server" Oct 30 21:08:40.443266 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.443253 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:08:40.443328 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.443266 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:08:40.443374 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.443333 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:08:40.443440 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.443413 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:08:40.443440 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.443422 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:08:40.443951 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.443930 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:08:40.444028 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.443989 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.485719 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.485691 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-149.ec2.internal\" not found" node="ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.520436 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.520406 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal"] Oct 30 21:08:40.520539 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.520472 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.521488 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.521471 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.521575 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.521504 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.521575 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.521515 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.523837 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.523825 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.523973 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.523959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.524025 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.523988 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.524650 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.524629 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.524650 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.524654 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.524775 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.524664 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.524775 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.524624 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.524775 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.524729 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.524775 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.524740 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.526812 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.526796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.526905 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.526819 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.527494 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.527477 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.527597 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.527504 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.527597 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.527516 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.543639 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.543624 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:08:40.544287 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.544271 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:08:40.544355 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.544303 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:08:40.544355 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.544318 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:08:40.544355 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.544343 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.562590 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.562575 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.562675 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.562597 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-149.ec2.internal\": node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.568110 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.568095 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-149.ec2.internal\" not found" node="ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.576569 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.576549 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-149.ec2.internal\" not found" node="ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.595941 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.595922 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.640933 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.640915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/524d7c023acdf7f01932ca99a8915366-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"524d7c023acdf7f01932ca99a8915366\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.641006 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.640936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.641006 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.640953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.696958 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.696906 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.741884 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.741863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.741951 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.741888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.741951 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.741903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/524d7c023acdf7f01932ca99a8915366-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"524d7c023acdf7f01932ca99a8915366\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.741951 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.741947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/524d7c023acdf7f01932ca99a8915366-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"524d7c023acdf7f01932ca99a8915366\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.742056 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.741964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.742056 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.741964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.798007 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.797981 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:40.874115 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.874082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.881417 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:40.881281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:40.898339 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:40.898318 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:41.002049 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:41.001982 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:41.102493 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:41.102459 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:41.170467 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.170436 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:08:41.170900 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.170577 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:08:41.202772 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:41.202742 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:08:41.239461 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.239438 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:08:41.339420 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.339342 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:08:41.366969 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.366947 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:08:41.369137 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.369122 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:08:41.387792 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.387769 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:08:41.400982 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.400930 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:03:41 +0000 UTC" deadline="2028-06-21 07:29:40.633986327 +0000 UTC" Oct 30 21:08:41.400982 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.400981 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14386h20m59.233008553s" Oct 30 21:08:41.461168 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.461140 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:08:41.485920 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:41.485888 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad7dad2adb3692daf79090c2b068e15.slice/crio-e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e WatchSource:0}: Error finding container e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e: Status 404 returned error can't find the container with id e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e Oct 30 21:08:41.486348 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:41.486322 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524d7c023acdf7f01932ca99a8915366.slice/crio-6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd WatchSource:0}: Error finding container 6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd: Status 404 returned error can't find the container with id 6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd Oct 30 21:08:41.491536 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.491519 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:08:41.815997 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:41.815918 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:08:42.211279 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.211196 2576 apiserver.go:52] "Watching apiserver" Oct 30 21:08:42.238921 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.238891 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:08:42.254209 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.254181 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:08:42.254620 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.254598 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-c7kgv","openshift-multus/multus-additional-cni-plugins-cf9xp","openshift-multus/network-metrics-daemon-c88gd","kube-system/konnectivity-agent-hsqfc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4","openshift-network-diagnostics/network-check-target-85c8w","openshift-ovn-kubernetes/ovnkube-node-psc8l","kube-system/global-pull-secret-syncer-b9l4r","kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal","openshift-image-registry/node-ca-87n87","openshift-multus/multus-kzg7s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal","openshift-network-operator/iptables-alerter-r6tmj","openshift-cluster-node-tuning-operator/tuned-s586p"] Oct 30 21:08:42.257457 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.257431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.260558 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.260533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.264930 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.264621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.264930 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.264705 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:42.267516 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.267494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.269596 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.269575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.271922 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.271902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:42.272027 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.271964 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:42.273860 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.273651 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.273860 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.273738 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:08:42.273860 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.273784 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:08:42.273860 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.273820 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:08:42.273860 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.273833 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.274489 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.274318 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d6fgf\"" Oct 30 21:08:42.275749 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.275723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.276166 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.276148 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:08:42.276482 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.276466 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.276679 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.276656 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:08:42.281528 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.279639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.281970 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.281954 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hnk74\"" Oct 30 21:08:42.284054 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.284033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.286313 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.286288 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:08:42.286411 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.286356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.286670 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.286361 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-msdvn" Oct 30 21:08:42.289925 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.289901 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-msdvn" Oct 30 21:08:42.290603 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.290581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.294098 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.294081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.294191 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.294151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:42.340906 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.340882 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:08:42.351403 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovn-node-metrics-cert\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.351553 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-cnibin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.351553 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw94c\" (UniqueName: \"kubernetes.io/projected/106e7cc6-df7c-4e5a-8876-1c0456a35023-kube-api-access-jw94c\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.351553 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-run\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.351695 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-tmp-dir\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.351695 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-iptables-alerter-script\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.351695 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cnl\" (UniqueName: \"kubernetes.io/projected/209912e2-5544-4a98-9282-ed7e28c449d0-kube-api-access-v4cnl\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.351807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-serviceca\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.351807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-netns\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.351807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-daemon-config\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.351917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.351917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-var-lib-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.351917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-log-socket\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.351917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-dbus\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.351917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-kubernetes\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbks\" (UniqueName: \"kubernetes.io/projected/b13507f7-c09f-46bd-b76b-50ba62300dfb-kube-api-access-tqbks\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2459def9-9723-4de1-9a93-5a2dfb7a3cca-agent-certs\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnsn\" (UniqueName: \"kubernetes.io/projected/95dba424-9ecd-43b9-a09e-8c643019e969-kube-api-access-gcnsn\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.351982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l656s\" (UniqueName: \"kubernetes.io/projected/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-kube-api-access-l656s\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbcm\" (UniqueName: \"kubernetes.io/projected/21966c11-af60-40d2-bde4-d6cb8296cad4-kube-api-access-tgbcm\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-env-overrides\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-kubelet\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysconfig\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-var-lib-kubelet\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-tuned\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-ovn\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-script-lib\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2459def9-9723-4de1-9a93-5a2dfb7a3cca-konnectivity-ca\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-system-cni-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-etc-kubernetes\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-systemd\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-lib-modules\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkt5\" (UniqueName: \"kubernetes.io/projected/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-kube-api-access-bvkt5\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-bin\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-config\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-modprobe-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2f2m\" (UniqueName: \"kubernetes.io/projected/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-kube-api-access-l2f2m\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.352607 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-hosts-file\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-cnibin\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-systemd-units\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-node-log\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-kubelet-config\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-kubelet\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-netd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-bin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-hostroot\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-sys\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-multus\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-conf-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.352991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-netns\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-system-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-socket-dir-parent\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-k8s-cni-cncf-io\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-slash\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-host\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlpj\" (UniqueName: \"kubernetes.io/projected/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-kube-api-access-bmlpj\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-conf\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-systemd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-etc-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.353822 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-cni-binary-copy\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-host-slash\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-host\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-device-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-multus-certs\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.354595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.353712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-tmp\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.383054 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383029 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:08:42.383324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383291 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:08:42.383408 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383378 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.383625 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383609 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.383701 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383620 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.383783 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383752 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.383990 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383951 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:08:42.384063 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384040 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:08:42.384122 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.383952 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.384122 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384099 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:08:42.384257 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384245 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.384311 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384296 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5899\"" Oct 30 21:08:42.384345 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384325 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjvtw\"" Oct 30 21:08:42.384378 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384350 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.384378 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384353 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:08:42.384477 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384385 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.384477 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384457 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t284x\"" Oct 30 21:08:42.384577 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384516 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:08:42.384577 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384551 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.384681 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384588 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:08:42.384681 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384644 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2tg5\"" Oct 30 21:08:42.384780 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384688 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:08:42.384780 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384755 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nxx4w\"" Oct 30 21:08:42.384780 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384299 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:08:42.384930 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384836 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8m2s\"" Oct 30 21:08:42.384930 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.384908 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jv27h\"" Oct 30 21:08:42.427812 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.427715 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" event={"ID":"524d7c023acdf7f01932ca99a8915366","Type":"ContainerStarted","Data":"6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd"} Oct 30 21:08:42.428952 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.428923 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"9ad7dad2adb3692daf79090c2b068e15","Type":"ContainerStarted","Data":"e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e"} Oct 30 21:08:42.454848 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-env-overrides\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-kubelet\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysconfig\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-var-lib-kubelet\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-tuned\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454967 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-kubelet\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.455003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.454983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-ovn\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-script-lib\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2459def9-9723-4de1-9a93-5a2dfb7a3cca-konnectivity-ca\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455042 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-var-lib-kubelet\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-system-cni-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-etc-kubernetes\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysconfig\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-systemd\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-lib-modules\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkt5\" (UniqueName: \"kubernetes.io/projected/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-kube-api-access-bvkt5\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-etc-kubernetes\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-bin\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-config\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-modprobe-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f2m\" (UniqueName: \"kubernetes.io/projected/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-kube-api-access-l2f2m\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.455366 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-hosts-file\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-cnibin\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-systemd-units\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-node-log\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455481 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-env-overrides\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-system-cni-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-kubelet-config\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-kubelet\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-netd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.455639 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-bin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-hostroot\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-sys\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.455707 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret podName:4142b823-6b14-49f2-bf14-e8231f9296f3 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:42.955686198 +0000 UTC m=+3.136606075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret") pod "global-pull-secret-syncer-b9l4r" (UID: "4142b823-6b14-49f2-bf14-e8231f9296f3") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455714 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-script-lib\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-sys\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.456240 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455775 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-ovn\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455775 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-kubelet-config\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455824 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-netd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2459def9-9723-4de1-9a93-5a2dfb7a3cca-konnectivity-ca\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-bin\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455890 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-bin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-kubelet\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455979 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-hostroot\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456012 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-systemd\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456095 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-lib-modules\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.457070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456122 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-multus\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-conf-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-netns\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-system-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-config\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-system-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-socket-dir-parent\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-k8s-cni-cncf-io\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456349 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-modprobe-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456366 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-systemd-units\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456411 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-conf-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.457807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-netns\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-multus\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.455895 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-cnibin\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-k8s-cni-cncf-io\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-slash\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-hosts-file\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456536 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-node-log\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456585 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-socket-dir-parent\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-host\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlpj\" (UniqueName: \"kubernetes.io/projected/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-kube-api-access-bmlpj\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456679 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-slash\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456693 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.456703 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456733 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-host\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:42.458614 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.456823 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:42.956807411 +0000 UTC m=+3.137727294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-conf\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-systemd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-etc-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.456979 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-systemd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-cni-binary-copy\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-etc-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-host-slash\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457091 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-host-slash\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-conf\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457096 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-host\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-host\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457169 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-device-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.459450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457211 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-multus-certs\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-multus-certs\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457250 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-device-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-tmp\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovn-node-metrics-cert\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-cnibin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw94c\" (UniqueName: \"kubernetes.io/projected/106e7cc6-df7c-4e5a-8876-1c0456a35023-kube-api-access-jw94c\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-run\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-tmp-dir\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-iptables-alerter-script\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cnl\" (UniqueName: \"kubernetes.io/projected/209912e2-5544-4a98-9282-ed7e28c449d0-kube-api-access-v4cnl\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-cnibin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-serviceca\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-netns\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.460324 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-daemon-config\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-var-lib-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-log-socket\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-dbus\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457634 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-run\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-kubernetes\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-log-socket\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbks\" (UniqueName: \"kubernetes.io/projected/b13507f7-c09f-46bd-b76b-50ba62300dfb-kube-api-access-tqbks\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2459def9-9723-4de1-9a93-5a2dfb7a3cca-agent-certs\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnsn\" (UniqueName: \"kubernetes.io/projected/95dba424-9ecd-43b9-a09e-8c643019e969-kube-api-access-gcnsn\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-tmp-dir\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l656s\" (UniqueName: \"kubernetes.io/projected/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-kube-api-access-l656s\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbcm\" (UniqueName: \"kubernetes.io/projected/21966c11-af60-40d2-bde4-d6cb8296cad4-kube-api-access-tgbcm\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-kubernetes\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.457600 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-cni-binary-copy\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458045 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-netns\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.461040 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-dbus\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458120 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-iptables-alerter-script\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458143 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-daemon-config\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-var-lib-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-serviceca\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.458715 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.460885 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-tmp\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.460966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovn-node-metrics-cert\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.461112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2459def9-9723-4de1-9a93-5a2dfb7a3cca-agent-certs\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.461591 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.461286 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-tuned\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.477884 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.477858 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:08:42.478021 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.477890 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:08:42.478021 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.477905 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncd4q for pod openshift-network-diagnostics/network-check-target-85c8w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:42.478021 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.478009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q podName:0c1e1f33-a5b9-442c-8589-970667a86890 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:42.977989051 +0000 UTC m=+3.158908949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ncd4q" (UniqueName: "kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q") pod "network-check-target-85c8w" (UID: "0c1e1f33-a5b9-442c-8589-970667a86890") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:42.479568 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.479541 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlpj\" (UniqueName: \"kubernetes.io/projected/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-kube-api-access-bmlpj\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.480227 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.480200 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f2m\" (UniqueName: \"kubernetes.io/projected/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-kube-api-access-l2f2m\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.480907 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.480876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkt5\" (UniqueName: \"kubernetes.io/projected/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-kube-api-access-bvkt5\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.481381 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.481354 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l656s\" (UniqueName: \"kubernetes.io/projected/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-kube-api-access-l656s\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.481478 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.481360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnsn\" (UniqueName: \"kubernetes.io/projected/95dba424-9ecd-43b9-a09e-8c643019e969-kube-api-access-gcnsn\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.482441 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.482399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cnl\" (UniqueName: \"kubernetes.io/projected/209912e2-5544-4a98-9282-ed7e28c449d0-kube-api-access-v4cnl\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.482814 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.482791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw94c\" (UniqueName: \"kubernetes.io/projected/106e7cc6-df7c-4e5a-8876-1c0456a35023-kube-api-access-jw94c\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.483207 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.483184 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbks\" (UniqueName: \"kubernetes.io/projected/b13507f7-c09f-46bd-b76b-50ba62300dfb-kube-api-access-tqbks\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.486434 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.486415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbcm\" (UniqueName: \"kubernetes.io/projected/21966c11-af60-40d2-bde4-d6cb8296cad4-kube-api-access-tgbcm\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.590279 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.590243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:08:42.592522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.592462 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:42.596784 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.596765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:08:42.602580 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:42.602554 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b62bf2e_5880_40f6_8d4b_ee60e2a8c30c.slice/crio-718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab WatchSource:0}: Error finding container 718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab: Status 404 returned error can't find the container with id 718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab Oct 30 21:08:42.604003 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.603977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:08:42.607826 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.607779 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:08:42.609543 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:42.609487 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2459def9_9723_4de1_9a93_5a2dfb7a3cca.slice/crio-73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50 WatchSource:0}: Error finding container 73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50: Status 404 returned error can't find the container with id 73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50 Oct 30 21:08:42.610702 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:42.610624 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13507f7_c09f_46bd_b76b_50ba62300dfb.slice/crio-8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2 WatchSource:0}: Error finding container 8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2: Status 404 returned error can't find the container with id 8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2 Oct 30 21:08:42.623949 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.623927 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:08:42.636518 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:42.636474 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bafe23a_4bb7_477d_81a1_6024f5c3d9d4.slice/crio-d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691 WatchSource:0}: Error finding container d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691: Status 404 returned error can't find the container with id d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691 Oct 30 21:08:42.644402 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.644375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-87n87" Oct 30 21:08:42.647632 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.647611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kzg7s" Oct 30 21:08:42.649552 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.649532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:08:42.655919 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:08:42.655889 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b8fafe_40b1_43de_b7f2_63a4f2cbfd0e.slice/crio-8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28 WatchSource:0}: Error finding container 8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28: Status 404 returned error can't find the container with id 8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28 Oct 30 21:08:42.693326 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.692879 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:08:42.960594 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.960499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:42.960752 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:42.960607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:42.960752 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.960686 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:42.960752 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.960737 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:42.960897 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.960767 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:43.960749171 +0000 UTC m=+4.141669066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:42.960897 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:42.960790 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret podName:4142b823-6b14-49f2-bf14-e8231f9296f3 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:43.960774432 +0000 UTC m=+4.141694327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret") pod "global-pull-secret-syncer-b9l4r" (UID: "4142b823-6b14-49f2-bf14-e8231f9296f3") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:43.061931 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.061891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:43.062084 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.062065 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:08:43.062129 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.062085 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:08:43.062129 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.062099 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncd4q for pod openshift-network-diagnostics/network-check-target-85c8w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:43.062193 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.062161 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q podName:0c1e1f33-a5b9-442c-8589-970667a86890 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:44.062142963 +0000 UTC m=+4.243062843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncd4q" (UniqueName: "kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q") pod "network-check-target-85c8w" (UID: "0c1e1f33-a5b9-442c-8589-970667a86890") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:43.293351 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.293234 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:03:43 +0000 UTC" deadline="2028-05-16 12:02:16.585455369 +0000 UTC" Oct 30 21:08:43.293351 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.293298 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13526h53m33.292161452s" Oct 30 21:08:43.421291 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.420734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:43.421291 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.420884 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:43.451165 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.451038 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" event={"ID":"209912e2-5544-4a98-9282-ed7e28c449d0","Type":"ContainerStarted","Data":"7d900c012506d9ce2649bfe5419204d063aafe5f25b90b291508da8e18db8a74"} Oct 30 21:08:43.466649 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.466618 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6tmj" event={"ID":"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c","Type":"ContainerStarted","Data":"718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab"} Oct 30 21:08:43.486367 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.486284 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerStarted","Data":"d7d9141568c42fde15f5e64a81efffa2b58b10ce4ef5d4a0213878a91069b1c5"} Oct 30 21:08:43.494068 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.493982 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kzg7s" event={"ID":"21966c11-af60-40d2-bde4-d6cb8296cad4","Type":"ContainerStarted","Data":"8b3525026d116424ba68c89e44306b992182460015bf1de7199480c9bde41f6c"} Oct 30 21:08:43.499665 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.499617 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-87n87" event={"ID":"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e","Type":"ContainerStarted","Data":"8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28"} Oct 30 21:08:43.513344 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.513310 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c7kgv" event={"ID":"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4","Type":"ContainerStarted","Data":"d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691"} Oct 30 21:08:43.518899 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.518871 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s586p" event={"ID":"ad5e0914-9ca0-4dbf-952e-1b9812908b1c","Type":"ContainerStarted","Data":"5d80667300274cbb1831e86848ba51682ae390478b391baf950ddbd16a18262c"} Oct 30 21:08:43.525018 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.524991 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2"} Oct 30 21:08:43.537212 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.537185 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hsqfc" event={"ID":"2459def9-9723-4de1-9a93-5a2dfb7a3cca","Type":"ContainerStarted","Data":"73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50"} Oct 30 21:08:43.975305 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.975025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:43.975533 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:43.975358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:43.975605 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.975569 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:43.975662 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.975616 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:43.975662 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.975642 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret podName:4142b823-6b14-49f2-bf14-e8231f9296f3 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:45.975621895 +0000 UTC m=+6.156541776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret") pod "global-pull-secret-syncer-b9l4r" (UID: "4142b823-6b14-49f2-bf14-e8231f9296f3") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:43.975662 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:43.975661 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:45.975651686 +0000 UTC m=+6.156571565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:44.076623 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.076536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:44.076797 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:44.076722 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:08:44.076797 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:44.076755 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:08:44.076797 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:44.076772 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncd4q for pod openshift-network-diagnostics/network-check-target-85c8w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:44.076951 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:44.076849 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q podName:0c1e1f33-a5b9-442c-8589-970667a86890 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:46.076826417 +0000 UTC m=+6.257746312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncd4q" (UniqueName: "kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q") pod "network-check-target-85c8w" (UID: "0c1e1f33-a5b9-442c-8589-970667a86890") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:44.293586 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.293500 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:03:43 +0000 UTC" deadline="2028-03-26 01:59:45.268510897 +0000 UTC" Oct 30 21:08:44.293586 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.293542 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12292h51m0.974973125s" Oct 30 21:08:44.422941 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.422862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:44.423100 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:44.423002 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:44.423459 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.423437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:44.423586 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:44.423564 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:44.547937 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.547896 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ad7dad2adb3692daf79090c2b068e15" containerID="501e50849ed3267c04a07d21b78ee1600a8319f2ed8a143c7327f2afbe420261" exitCode=0 Oct 30 21:08:44.548124 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.547988 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"9ad7dad2adb3692daf79090c2b068e15","Type":"ContainerDied","Data":"501e50849ed3267c04a07d21b78ee1600a8319f2ed8a143c7327f2afbe420261"} Oct 30 21:08:44.563491 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:44.563021 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" event={"ID":"524d7c023acdf7f01932ca99a8915366","Type":"ContainerStarted","Data":"096528ed2f68c12793386d1ca1d901949a308f5c6eae25709168f8cb79e887f8"} Oct 30 21:08:45.420715 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:45.420629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:45.421225 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:45.420786 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:45.575108 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:45.574977 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"9ad7dad2adb3692daf79090c2b068e15","Type":"ContainerStarted","Data":"08451d328bf52377b9bedef01e5778f53bc50649a1e17684218f7276391d90c7"} Oct 30 21:08:45.613403 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:45.612403 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" podStartSLOduration=3.612383734 podStartE2EDuration="3.612383734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:44.629787198 +0000 UTC m=+4.810707099" watchObservedRunningTime="2026-10-30 21:08:45.612383734 +0000 UTC m=+5.793303634" Oct 30 21:08:45.995378 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:45.995189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:45.995547 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:45.995347 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:45.995547 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:45.995451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:45.995547 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:45.995484 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret podName:4142b823-6b14-49f2-bf14-e8231f9296f3 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:49.995463451 +0000 UTC m=+10.176383335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret") pod "global-pull-secret-syncer-b9l4r" (UID: "4142b823-6b14-49f2-bf14-e8231f9296f3") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:45.995547 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:45.995532 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:45.995756 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:45.995569 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:49.995556823 +0000 UTC m=+10.176476700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:46.095962 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:46.095922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:46.096212 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:46.096178 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:08:46.096212 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:46.096204 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:08:46.096339 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:46.096217 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncd4q for pod openshift-network-diagnostics/network-check-target-85c8w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:46.096339 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:46.096279 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q podName:0c1e1f33-a5b9-442c-8589-970667a86890 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:50.096261315 +0000 UTC m=+10.277181207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncd4q" (UniqueName: "kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q") pod "network-check-target-85c8w" (UID: "0c1e1f33-a5b9-442c-8589-970667a86890") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:46.420711 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:46.420632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:46.420873 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:46.420762 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:46.421309 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:46.421154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:46.421309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:46.421246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:46.586084 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:46.584540 2576 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="517b6558187902b51db47c16375ce934b222711465d832f33e82ac48002fc61b" exitCode=0 Oct 30 21:08:46.586084 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:46.585786 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerDied","Data":"517b6558187902b51db47c16375ce934b222711465d832f33e82ac48002fc61b"} Oct 30 21:08:46.646529 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:46.646466 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" podStartSLOduration=4.646426552 podStartE2EDuration="4.646426552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:45.613273577 +0000 UTC m=+5.794193479" watchObservedRunningTime="2026-10-30 21:08:46.646426552 +0000 UTC m=+6.827346454" Oct 30 21:08:47.420729 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:47.420515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:47.420729 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:47.420695 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:47.592084 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:47.591264 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hsqfc" event={"ID":"2459def9-9723-4de1-9a93-5a2dfb7a3cca","Type":"ContainerStarted","Data":"799273109cfe7f801b9772e1af076746fb3f530104a38fd880eb0bcc4364fe3b"} Oct 30 21:08:48.423756 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:48.422921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:48.423756 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:48.423081 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:48.423756 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:48.423541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:48.423756 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:48.423650 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:48.598898 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:48.598725 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-87n87" event={"ID":"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e","Type":"ContainerStarted","Data":"1c0519e509476ba2fad306e6c3995fba88ba81255074d4ca4fd21294ceaa76c4"} Oct 30 21:08:48.637109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:48.636278 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-87n87" podStartSLOduration=2.522433833 podStartE2EDuration="7.636262235s" totalImagesPullingTime="5.113828402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:48.636125633 +0000 UTC m=+8.817045534" watchObservedRunningTime="2026-10-30 21:08:48.636262235 +0000 UTC m=+8.817182135" Oct 30 21:08:48.637109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:48.636786 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hsqfc" podStartSLOduration=3.651752525 podStartE2EDuration="7.636769335s" totalImagesPullingTime="3.98501681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:47.630895572 +0000 UTC m=+7.811815474" watchObservedRunningTime="2026-10-30 21:08:48.636769335 +0000 UTC m=+8.817689238" Oct 30 21:08:49.350686 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:49.350652 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:49.351350 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:49.351320 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:08:49.420300 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:49.420252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:49.420496 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:49.420438 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:49.604129 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:49.604047 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" event={"ID":"209912e2-5544-4a98-9282-ed7e28c449d0","Type":"ContainerStarted","Data":"7c2179f6b54ba9f3d3387508b6b04ad58b2e9c2ea805641a1cf2b55f2dbbcbdf"} Oct 30 21:08:50.026259 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.026162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:50.026259 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.026239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:50.026525 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.026420 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:50.026525 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.026509 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:58.026486039 +0000 UTC m=+18.207405931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:50.026639 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.026589 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:50.026639 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.026630 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret podName:4142b823-6b14-49f2-bf14-e8231f9296f3 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:58.02661701 +0000 UTC m=+18.207536902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret") pod "global-pull-secret-syncer-b9l4r" (UID: "4142b823-6b14-49f2-bf14-e8231f9296f3") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:50.127504 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.126762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:50.127504 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.126959 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:08:50.127504 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.126987 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:08:50.127504 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.127002 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncd4q for pod openshift-network-diagnostics/network-check-target-85c8w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:50.127504 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.127073 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q podName:0c1e1f33-a5b9-442c-8589-970667a86890 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:58.127051938 +0000 UTC m=+18.307971821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncd4q" (UniqueName: "kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q") pod "network-check-target-85c8w" (UID: "0c1e1f33-a5b9-442c-8589-970667a86890") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:50.421562 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.421530 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:50.421741 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.421572 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:50.421741 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.421648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:50.422028 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:50.422008 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:50.452092 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.452032 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:08:50.611381 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.611356 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:08:50.611837 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:50.611805 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" event={"ID":"209912e2-5544-4a98-9282-ed7e28c449d0","Type":"ContainerStarted","Data":"c8c04ac5cc7d468765c594fa2e1fff459bf9bc1c4691b6f63fd82116d1dfc0d5"} Oct 30 21:08:51.427409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.420494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:51.427409 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:51.420654 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:51.451619 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.451458 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:08:50.452055404Z","UUID":"9c57eeef-cd81-4171-b190-596666844f50","Handler":null,"Name":"","Endpoint":""} Oct 30 21:08:51.455539 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.455516 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:08:51.455539 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.455544 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:08:51.618894 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.618860 2576 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="147d743c9f41c00f0ac1a0d285fb94cfe6464a7f8e340ff47b5e99f1d5ab928d" exitCode=0 Oct 30 21:08:51.619319 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.618956 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerDied","Data":"147d743c9f41c00f0ac1a0d285fb94cfe6464a7f8e340ff47b5e99f1d5ab928d"} Oct 30 21:08:51.623022 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.621760 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kzg7s" event={"ID":"21966c11-af60-40d2-bde4-d6cb8296cad4","Type":"ContainerStarted","Data":"fafe417ae68ab73348b709394f06d478c62cd6d01c85899cb0199479df71b65e"} Oct 30 21:08:51.625062 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.625036 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c7kgv" event={"ID":"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4","Type":"ContainerStarted","Data":"44e7f0d97c09d68feadb646f2c9fc65bd33d1bd2e85dc8733b581b7268d901ee"} Oct 30 21:08:51.696595 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:51.696289 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c7kgv" podStartSLOduration=2.423965751 podStartE2EDuration="10.696268742s" totalImagesPullingTime="8.272302991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:51.695266445 +0000 UTC m=+11.876186348" watchObservedRunningTime="2026-10-30 21:08:51.696268742 +0000 UTC m=+11.877188642" Oct 30 21:08:52.420000 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.419967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:52.420276 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:52.420148 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:52.420807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.420787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:52.420958 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:52.420929 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:52.630189 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.630100 2576 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="486b18bee5f348d0a6ac9a360f105f9ba3c6b2f02542eba8f928880a89219080" exitCode=0 Oct 30 21:08:52.630687 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.630206 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerDied","Data":"486b18bee5f348d0a6ac9a360f105f9ba3c6b2f02542eba8f928880a89219080"} Oct 30 21:08:52.634587 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.634559 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" event={"ID":"209912e2-5544-4a98-9282-ed7e28c449d0","Type":"ContainerStarted","Data":"d8b323e05f0f18c33973991ca419cb53ed0f16fe4c49be04e03a8b7c5fffe088"} Oct 30 21:08:52.636919 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.636892 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6tmj" event={"ID":"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c","Type":"ContainerStarted","Data":"0141a3b69a1e20a054889848834ee76a2b25058c409186a864f1af85f806ebbf"} Oct 30 21:08:52.670725 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.670682 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kzg7s" podStartSLOduration=3.812486565 podStartE2EDuration="11.670669355s" totalImagesPullingTime="7.85818279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:51.732926917 +0000 UTC m=+11.913846856" watchObservedRunningTime="2026-10-30 21:08:52.670669355 +0000 UTC m=+12.851589254" Oct 30 21:08:52.706933 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.706878 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6tmj" podStartSLOduration=3.746941406 podStartE2EDuration="11.706861626s" totalImagesPullingTime="7.95992022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:52.706774099 +0000 UTC m=+12.887694003" watchObservedRunningTime="2026-10-30 21:08:52.706861626 +0000 UTC m=+12.887781529" Oct 30 21:08:52.745207 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:52.745145 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" podStartSLOduration=3.211925198 podStartE2EDuration="11.745127604s" totalImagesPullingTime="8.533202406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:52.744681918 +0000 UTC m=+12.925601820" watchObservedRunningTime="2026-10-30 21:08:52.745127604 +0000 UTC m=+12.926047507" Oct 30 21:08:53.420055 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:53.420029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:53.420161 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:53.420148 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:53.641922 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:53.641890 2576 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="890d117ec1b9c614be27533c3a4f2c4307ce00d193b1e1c76e8ee398d626c778" exitCode=0 Oct 30 21:08:53.642538 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:53.641978 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerDied","Data":"890d117ec1b9c614be27533c3a4f2c4307ce00d193b1e1c76e8ee398d626c778"} Oct 30 21:08:53.643763 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:53.643735 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s586p" event={"ID":"ad5e0914-9ca0-4dbf-952e-1b9812908b1c","Type":"ContainerStarted","Data":"e01ed87423201c24fb409414ac450aa22168d55f9719091beca7173b314f8e12"} Oct 30 21:08:54.420465 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:54.420425 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:54.420615 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:54.420425 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:54.420615 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:54.420569 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:54.420916 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:54.420869 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:55.420536 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:55.420508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:55.421127 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:55.420646 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:55.651342 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:55.651273 2576 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="00f98d9c6dc7a0b88c9d9d50e28417046e023370c1ac43960d6850fef7f0b32e" exitCode=0 Oct 30 21:08:55.651342 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:55.651334 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerDied","Data":"00f98d9c6dc7a0b88c9d9d50e28417046e023370c1ac43960d6850fef7f0b32e"} Oct 30 21:08:55.715144 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:55.714910 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s586p" podStartSLOduration=4.286036246 podStartE2EDuration="14.714889637s" totalImagesPullingTime="10.428853391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:53.714887664 +0000 UTC m=+13.895807560" watchObservedRunningTime="2026-10-30 21:08:55.714889637 +0000 UTC m=+15.895809536" Oct 30 21:08:56.420510 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:56.420476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:56.420662 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:56.420607 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:56.420977 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:56.420464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:56.421055 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:56.421035 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:56.656836 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:56.656801 2576 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="a2786a39b0fdb26003e15b6ca2e2e49fac71bf0a901d7226f670f2c079019a03" exitCode=0 Oct 30 21:08:56.656997 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:56.656862 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerDied","Data":"a2786a39b0fdb26003e15b6ca2e2e49fac71bf0a901d7226f670f2c079019a03"} Oct 30 21:08:57.420421 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.420397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:57.420572 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:57.420538 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:57.663025 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.662993 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" event={"ID":"95dba424-9ecd-43b9-a09e-8c643019e969","Type":"ContainerStarted","Data":"cfcfa10fa863e74c3737f412ef99e51d14d1f45f85710f4a9dbd90c51b763d2f"} Oct 30 21:08:57.665806 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.665782 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"afb3474e96a61009d996cf6d4f30c115cbcc4701e7da9165cd3348491b0f126d"} Oct 30 21:08:57.665914 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.665841 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"dc989b399692e043b5aef9b542e8dcde05f4eb4721d19bd6ed39f1bb66b0621f"} Oct 30 21:08:57.665914 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.665871 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"76c0d7d3bb00a9f949466768333ef7f3d5ef808be003f81edb7ff85f69f631ca"} Oct 30 21:08:57.665914 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.665899 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"71607e0f1da98d62b248f1f80507de6c55976b548228027564d70f90275c3f58"} Oct 30 21:08:57.666011 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.665924 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"63ea3766aa64b6f841af406556b675c38b84a211b8392ae52dbef26c66db664a"} Oct 30 21:08:57.666011 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.665952 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"3917edca880acde7efc4a1e1aaec69532511aae21a3cff5c1f2f8d601fdcb386"} Oct 30 21:08:57.728585 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:57.728481 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cf9xp" podStartSLOduration=6.052630695 podStartE2EDuration="16.728464191s" totalImagesPullingTime="10.675833496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:57.728269033 +0000 UTC m=+17.909188929" watchObservedRunningTime="2026-10-30 21:08:57.728464191 +0000 UTC m=+17.909384089" Oct 30 21:08:58.086177 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:58.085943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:58.086177 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:58.086143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:58.086353 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.086097 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:58.086353 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.086230 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:58.086353 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.086248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret podName:4142b823-6b14-49f2-bf14-e8231f9296f3 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:14.086233204 +0000 UTC m=+34.267153080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret") pod "global-pull-secret-syncer-b9l4r" (UID: "4142b823-6b14-49f2-bf14-e8231f9296f3") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:08:58.086353 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.086266 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:14.0862564 +0000 UTC m=+34.267176276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:08:58.187086 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:58.187050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:58.187227 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.187166 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:08:58.187227 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.187179 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:08:58.187227 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.187191 2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncd4q for pod openshift-network-diagnostics/network-check-target-85c8w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:58.187341 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.187242 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q podName:0c1e1f33-a5b9-442c-8589-970667a86890 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:14.187226621 +0000 UTC m=+34.368146502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncd4q" (UniqueName: "kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q") pod "network-check-target-85c8w" (UID: "0c1e1f33-a5b9-442c-8589-970667a86890") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:08:58.419846 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:58.419813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:08:58.419992 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.419931 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:08:58.420049 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:58.419988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:08:58.420140 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:58.420114 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:08:59.420071 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:59.420043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:08:59.420464 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:08:59.420138 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:08:59.672695 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:08:59.672623 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"a3c72909b07ec52a0e55a150905afcfda40b513763597ac3c6deb7d5d750be69"} Oct 30 21:09:00.421226 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:00.421185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:00.421781 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:00.421275 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:09:00.421781 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:00.421356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:00.421781 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:00.421453 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:09:01.420637 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:01.420590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:01.420816 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:01.420705 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:09:02.419708 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:02.419677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:02.420457 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:02.419789 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:09:02.420457 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:02.419847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:02.420457 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:02.419930 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:09:02.682294 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:02.682214 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" event={"ID":"b13507f7-c09f-46bd-b76b-50ba62300dfb","Type":"ContainerStarted","Data":"f4b2d76f9be95edeed2e3cd5eadeeb65ce4757e4851e39df4fa46cee4e6ced2f"} Oct 30 21:09:02.682529 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:02.682503 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:02.697269 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:02.697244 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:02.764211 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:02.764169 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" podStartSLOduration=7.159318943 podStartE2EDuration="21.764156462s" totalImagesPullingTime="14.604837519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:02.763588874 +0000 UTC m=+22.944508785" watchObservedRunningTime="2026-10-30 21:09:02.764156462 +0000 UTC m=+22.945076425" Oct 30 21:09:03.420313 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.420271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:03.420798 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:03.420475 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:09:03.685205 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.685126 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:03.685357 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.685232 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:03.701365 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.701341 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:03.772507 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.772475 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b9l4r"] Oct 30 21:09:03.772672 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.772627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:03.772755 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:03.772732 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:09:03.776045 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.776017 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-85c8w"] Oct 30 21:09:03.776175 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.776102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:03.776232 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:03.776188 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:09:03.776576 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.776554 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c88gd"] Oct 30 21:09:03.776671 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:03.776645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:03.776743 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:03.776724 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:09:05.420273 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:05.420083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:05.420790 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:05.420140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:05.420790 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:05.420344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:09:05.420790 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:05.420140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:05.420790 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:05.420473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:09:05.420790 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:05.420566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:09:07.420281 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:07.420253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:07.420713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:07.420253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:07.420713 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:07.420359 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85c8w" podUID="0c1e1f33-a5b9-442c-8589-970667a86890" Oct 30 21:09:07.420713 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:07.420453 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c88gd" podUID="106e7cc6-df7c-4e5a-8876-1c0456a35023" Oct 30 21:09:07.420713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:07.420252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:07.420713 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:07.420528 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b9l4r" podUID="4142b823-6b14-49f2-bf14-e8231f9296f3" Oct 30 21:09:08.530405 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:08.530362 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:08.530980 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:08.530537 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:08.531046 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:08.531007 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:09.155515 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.155490 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeReady" Oct 30 21:09:09.155665 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.155598 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:09:09.201919 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.201857 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bnjl2"] Oct 30 21:09:09.205189 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.204802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:09.205525 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.205492 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j9ct9"] Oct 30 21:09:09.208320 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.208303 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kmzkx"] Oct 30 21:09:09.208439 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.208426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:09.210998 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.210982 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b65c6946-57jbj"] Oct 30 21:09:09.211115 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.211103 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:09.213804 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.213785 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xr594"] Oct 30 21:09:09.213949 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.213931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.218429 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.218218 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn"] Oct 30 21:09:09.218429 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.218285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.221290 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.221272 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gh745"] Oct 30 21:09:09.221446 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.221380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.224807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.224790 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:09:09.224962 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.224905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.225443 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.225125 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.225443 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.225161 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.227666 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.227641 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc"] Oct 30 21:09:09.227778 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.227760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.231274 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.231256 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vws6h"] Oct 30 21:09:09.231377 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.231361 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:09.234411 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.234239 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:09:09.239109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.235999 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:09:09.239109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.236366 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:09:09.239109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.236425 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:09:09.239109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.236660 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g2wf2\"" Oct 30 21:09:09.239109 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.236720 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mcwp8\"" Oct 30 21:09:09.241021 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.241000 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f"] Oct 30 21:09:09.241126 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.241071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.244454 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.244435 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w"] Oct 30 21:09:09.245966 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.245165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.248551 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.248531 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch"] Oct 30 21:09:09.248662 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.248642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.251885 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.251868 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-phllg"] Oct 30 21:09:09.251999 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.251982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:09.259502 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.259478 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77b4b6cbf9-7xzk8"] Oct 30 21:09:09.262108 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.260148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.262922 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.262902 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rp8hx"] Oct 30 21:09:09.263407 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.263369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.267409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.266811 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5lcrb"] Oct 30 21:09:09.267409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.267129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.269141 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269253 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269253 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269363 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwc6h\" (UniqueName: \"kubernetes.io/projected/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75-kube-api-access-zwc6h\") pod \"kas-connection-checker-685895c7c9-j9ct9\" (UID: \"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75\") " pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:09.269435 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkscn\" (UniqueName: \"kubernetes.io/projected/d9781bb5-7481-4803-8ef9-2eca6954c446-kube-api-access-bkscn\") pod \"kas-connection-checker-685895c7c9-bnjl2\" (UID: \"d9781bb5-7481-4803-8ef9-2eca6954c446\") " pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:09.269495 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269495 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269578 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269578 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dgl\" (UniqueName: \"kubernetes.io/projected/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe-kube-api-access-c4dgl\") pod \"kas-connection-checker-685895c7c9-kmzkx\" (UID: \"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe\") " pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:09.269643 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.269643 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.269639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.273518 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273498 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bnjl2"] Oct 30 21:09:09.273620 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273536 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j9ct9"] Oct 30 21:09:09.273620 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273559 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kmzkx"] Oct 30 21:09:09.273620 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273580 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc"] Oct 30 21:09:09.273620 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273604 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xr594"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273625 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273652 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273673 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-phllg"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273695 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273722 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rp8hx"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273750 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vws6h"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273771 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77b4b6cbf9-7xzk8"] Oct 30 21:09:09.273807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273791 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b65c6946-57jbj"] Oct 30 21:09:09.274044 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273815 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f"] Oct 30 21:09:09.274044 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273837 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gh745"] Oct 30 21:09:09.274044 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273858 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch"] Oct 30 21:09:09.274044 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.273870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.280710 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.280690 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5lcrb"] Oct 30 21:09:09.322556 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322530 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.322645 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322581 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:09:09.322645 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322624 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:09:09.322725 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322691 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:09:09.322770 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322725 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:09:09.322770 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322739 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.322865 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322773 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.322865 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322787 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.322865 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.322787 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323052 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323036 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:09:09.323141 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323125 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323306 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323289 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323312 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323362 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:09:09.323524 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323421 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323524 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323426 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323524 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323472 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323524 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323476 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323524 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323478 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323535 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323568 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323572 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323608 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323631 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323643 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323652 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323683 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.323982 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.323778 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:09:09.335616 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335597 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:09:09.335713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335615 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:09:09.335713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335673 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:09:09.335713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335688 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:09:09.335713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335703 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335741 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335781 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335785 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335822 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335821 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335837 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:09:09.335910 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335902 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:09:09.336163 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335946 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7hhvb\"" Oct 30 21:09:09.336163 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.335950 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-z65n7\"" Oct 30 21:09:09.336163 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336115 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6qvgx\"" Oct 30 21:09:09.336278 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336216 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w6j9l\"" Oct 30 21:09:09.336329 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336295 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fs65v\"" Oct 30 21:09:09.336364 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336335 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qbpqm\"" Oct 30 21:09:09.336429 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336381 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d8vcg\"" Oct 30 21:09:09.336489 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336474 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xsb5g\"" Oct 30 21:09:09.336577 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336565 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gtfcx\"" Oct 30 21:09:09.336637 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336607 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5rj44\"" Oct 30 21:09:09.336720 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.336707 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42gzh\"" Oct 30 21:09:09.348376 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.348356 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:09:09.348376 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.348378 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:09:09.348507 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.348414 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:09:09.370483 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9931cd13-1775-4f3d-9300-d2cb16df7acf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.370537 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5abe8e87-10f7-453b-872f-ca366598dd92-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.370537 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.370610 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9931cd13-1775-4f3d-9300-d2cb16df7acf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.370649 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-service-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.370678 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f71a2be6-1f45-43a6-ae97-8ebd007341fd-nginx-conf\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.370709 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.370794 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08beba50-863a-4f30-a1ad-324cfde2159f-serving-cert\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.370880 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.370880 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-trusted-ca\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.370880 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a877cc0-db2f-42fc-9057-a62433401379-serving-cert\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.370880 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-default-certificate\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.371065 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.371065 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-tmp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.371065 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.370975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.371065 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.371241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.371241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.371241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.371241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwg9v\" (UniqueName: \"kubernetes.io/projected/9931cd13-1775-4f3d-9300-d2cb16df7acf-kube-api-access-mwg9v\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.371241 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-tmp-dir\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.371450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkscn\" (UniqueName: \"kubernetes.io/projected/d9781bb5-7481-4803-8ef9-2eca6954c446-kube-api-access-bkscn\") pod \"kas-connection-checker-685895c7c9-bnjl2\" (UID: \"d9781bb5-7481-4803-8ef9-2eca6954c446\") " pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:09.371450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.371450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.371450 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a877cc0-db2f-42fc-9057-a62433401379-config\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.371574 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-config-volume\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.371574 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.371574 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-config\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.371574 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.371574 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-snapshots\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.371776 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.371776 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.371776 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dj8\" (UniqueName: \"kubernetes.io/projected/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-kube-api-access-v2dj8\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.371901 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmlw\" (UniqueName: \"kubernetes.io/projected/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-kube-api-access-4wmlw\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.371901 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.371901 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7bn\" (UniqueName: \"kubernetes.io/projected/8a877cc0-db2f-42fc-9057-a62433401379-kube-api-access-7x7bn\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.371901 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-stats-auth\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.372037 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zh2\" (UniqueName: \"kubernetes.io/projected/7c7fd700-057c-4ceb-88c4-01620fa6329d-kube-api-access-q4zh2\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.372037 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.371975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjnp\" (UniqueName: \"kubernetes.io/projected/08beba50-863a-4f30-a1ad-324cfde2159f-kube-api-access-bwjnp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.372037 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jfvg\" (UniqueName: \"kubernetes.io/projected/2b8386a1-1b52-440d-b134-9c42390f70ba-kube-api-access-5jfvg\") pod \"volume-data-source-validator-58c9f5f46d-48pxc\" (UID: \"2b8386a1-1b52-440d-b134-9c42390f70ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8rm\" (UniqueName: \"kubernetes.io/projected/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-kube-api-access-gm8rm\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xhlv\" (UniqueName: \"kubernetes.io/projected/ee11341a-54bb-412f-ad81-2c8ef689247a-kube-api-access-2xhlv\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.372137 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.372154 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65c6946-57jbj: secret "image-registry-tls" not found Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.372173 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee11341a-54bb-412f-ad81-2c8ef689247a-cert\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.372216 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls podName:dccf83b0-6149-4f2b-b3b1-db339a5a245e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.872200012 +0000 UTC m=+30.053119887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls") pod "image-registry-5b65c6946-57jbj" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e") : secret "image-registry-tls" not found Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc6h\" (UniqueName: \"kubernetes.io/projected/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75-kube-api-access-zwc6h\") pod \"kas-connection-checker-685895c7c9-j9ct9\" (UID: \"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75\") " pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dgl\" (UniqueName: \"kubernetes.io/projected/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe-kube-api-access-c4dgl\") pod \"kas-connection-checker-685895c7c9-kmzkx\" (UID: \"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe\") " pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:09.372549 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qwjr\" (UniqueName: \"kubernetes.io/projected/5abe8e87-10f7-453b-872f-ca366598dd92-kube-api-access-7qwjr\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.372917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-serving-cert\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.372917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.372917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372604 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.372917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.372917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34-kube-api-access-wmf2d\") pod \"network-check-source-6f78d59f99-dq7ch\" (UID: \"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:09.372917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.372907 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.375094 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.375073 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.375169 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.375078 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.391051 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.390191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.391568 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.391543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.392381 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.391882 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkscn\" (UniqueName: \"kubernetes.io/projected/d9781bb5-7481-4803-8ef9-2eca6954c446-kube-api-access-bkscn\") pod \"kas-connection-checker-685895c7c9-bnjl2\" (UID: \"d9781bb5-7481-4803-8ef9-2eca6954c446\") " pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:09.393653 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.393632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dgl\" (UniqueName: \"kubernetes.io/projected/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe-kube-api-access-c4dgl\") pod \"kas-connection-checker-685895c7c9-kmzkx\" (UID: \"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe\") " pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:09.393896 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.393878 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwc6h\" (UniqueName: \"kubernetes.io/projected/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75-kube-api-access-zwc6h\") pod \"kas-connection-checker-685895c7c9-j9ct9\" (UID: \"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75\") " pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:09.420292 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.420266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:09.420381 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.420365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:09.420456 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.420401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:09.434402 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.434327 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:09:09.434541 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.434526 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\"" Oct 30 21:09:09.434615 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.434584 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\"" Oct 30 21:09:09.434660 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.434647 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:09:09.434717 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.434705 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\"" Oct 30 21:09:09.473647 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjnp\" (UniqueName: \"kubernetes.io/projected/08beba50-863a-4f30-a1ad-324cfde2159f-kube-api-access-bwjnp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.473647 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.473647 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jfvg\" (UniqueName: \"kubernetes.io/projected/2b8386a1-1b52-440d-b134-9c42390f70ba-kube-api-access-5jfvg\") pod \"volume-data-source-validator-58c9f5f46d-48pxc\" (UID: \"2b8386a1-1b52-440d-b134-9c42390f70ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8rm\" (UniqueName: \"kubernetes.io/projected/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-kube-api-access-gm8rm\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xhlv\" (UniqueName: \"kubernetes.io/projected/ee11341a-54bb-412f-ad81-2c8ef689247a-kube-api-access-2xhlv\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee11341a-54bb-412f-ad81-2c8ef689247a-cert\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwjr\" (UniqueName: \"kubernetes.io/projected/5abe8e87-10f7-453b-872f-ca366598dd92-kube-api-access-7qwjr\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.473851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-serving-cert\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34-kube-api-access-wmf2d\") pod \"network-check-source-6f78d59f99-dq7ch\" (UID: \"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9931cd13-1775-4f3d-9300-d2cb16df7acf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.473964 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.473982 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5774b95-sq55b: secret "image-registry-tls" not found Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.474045 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls podName:57293ead-fcc2-47e7-8033-d0beec008c51 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.974029067 +0000 UTC m=+30.154948957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls") pod "image-registry-66d5774b95-sq55b" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51") : secret "image-registry-tls" not found Oct 30 21:09:09.474167 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.473963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5abe8e87-10f7-453b-872f-ca366598dd92-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9931cd13-1775-4f3d-9300-d2cb16df7acf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-service-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f71a2be6-1f45-43a6-ae97-8ebd007341fd-nginx-conf\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08beba50-863a-4f30-a1ad-324cfde2159f-serving-cert\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-trusted-ca\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a877cc0-db2f-42fc-9057-a62433401379-serving-cert\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-default-certificate\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-tmp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.474522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwg9v\" (UniqueName: \"kubernetes.io/projected/9931cd13-1775-4f3d-9300-d2cb16df7acf-kube-api-access-mwg9v\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-tmp-dir\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a877cc0-db2f-42fc-9057-a62433401379-config\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5abe8e87-10f7-453b-872f-ca366598dd92-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-config-volume\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-config\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-snapshots\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dj8\" (UniqueName: \"kubernetes.io/projected/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-kube-api-access-v2dj8\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.475058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmlw\" (UniqueName: \"kubernetes.io/projected/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-kube-api-access-4wmlw\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-service-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.474991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7bn\" (UniqueName: \"kubernetes.io/projected/8a877cc0-db2f-42fc-9057-a62433401379-kube-api-access-7x7bn\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-stats-auth\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zh2\" (UniqueName: \"kubernetes.io/projected/7c7fd700-057c-4ceb-88c4-01620fa6329d-kube-api-access-q4zh2\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f71a2be6-1f45-43a6-ae97-8ebd007341fd-nginx-conf\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.474723 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.475331 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.975314782 +0000 UTC m=+30.156234676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.474784 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.475374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.975363511 +0000 UTC m=+30.156283392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : secret "router-metrics-certs-default" not found Oct 30 21:09:09.475824 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475835 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-trusted-ca\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.475911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-config-volume\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.475931 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.975916135 +0000 UTC m=+30.156836026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : configmap references non-existent config key: service-ca.crt Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.476006 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.476048 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert podName:f71a2be6-1f45-43a6-ae97-8ebd007341fd nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.976033417 +0000 UTC m=+30.156953317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-phllg" (UID: "f71a2be6-1f45-43a6-ae97-8ebd007341fd") : secret "networking-console-plugin-cert" not found Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.476182 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.476272 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:09.476415 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.476329 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls podName:5abe8e87-10f7-453b-872f-ca366598dd92 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.976313304 +0000 UTC m=+30.157233196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xr594" (UID: "5abe8e87-10f7-453b-872f-ca366598dd92") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:09.476807 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.476543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-config\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.476863 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.476830 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.476912 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.476857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-tmp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.476961 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.476950 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:09:09.477008 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.476993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls podName:7c5b111d-8074-4bda-8ac9-7d1f1f623c59 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.976979002 +0000 UTC m=+30.157898883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls") pod "dns-default-rp8hx" (UID: "7c5b111d-8074-4bda-8ac9-7d1f1f623c59") : secret "dns-default-metrics-tls" not found Oct 30 21:09:09.477238 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.477214 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-snapshots\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.477315 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.477272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-tmp-dir\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.477472 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.477452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9931cd13-1775-4f3d-9300-d2cb16df7acf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.477793 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.477772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.478059 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.478037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a877cc0-db2f-42fc-9057-a62433401379-config\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.478142 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.478060 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08beba50-863a-4f30-a1ad-324cfde2159f-serving-cert\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.478519 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.478494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9931cd13-1775-4f3d-9300-d2cb16df7acf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.478640 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.478620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.479028 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.478993 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-serving-cert\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.479364 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.479341 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.479471 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.479450 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee11341a-54bb-412f-ad81-2c8ef689247a-cert\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.479640 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.479624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a877cc0-db2f-42fc-9057-a62433401379-serving-cert\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.480057 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.480039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-default-certificate\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.480344 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.480325 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-stats-auth\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.493643 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.493617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8rm\" (UniqueName: \"kubernetes.io/projected/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-kube-api-access-gm8rm\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.493809 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.493789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.494006 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.493985 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjnp\" (UniqueName: \"kubernetes.io/projected/08beba50-863a-4f30-a1ad-324cfde2159f-kube-api-access-bwjnp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.495463 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.495436 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwg9v\" (UniqueName: \"kubernetes.io/projected/9931cd13-1775-4f3d-9300-d2cb16df7acf-kube-api-access-mwg9v\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.495783 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.495759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zh2\" (UniqueName: \"kubernetes.io/projected/7c7fd700-057c-4ceb-88c4-01620fa6329d-kube-api-access-q4zh2\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.496618 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.496596 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jfvg\" (UniqueName: \"kubernetes.io/projected/2b8386a1-1b52-440d-b134-9c42390f70ba-kube-api-access-5jfvg\") pod \"volume-data-source-validator-58c9f5f46d-48pxc\" (UID: \"2b8386a1-1b52-440d-b134-9c42390f70ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:09.496814 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.496796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7bn\" (UniqueName: \"kubernetes.io/projected/8a877cc0-db2f-42fc-9057-a62433401379-kube-api-access-7x7bn\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.496951 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.496931 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34-kube-api-access-wmf2d\") pod \"network-check-source-6f78d59f99-dq7ch\" (UID: \"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:09.497224 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.497206 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwjr\" (UniqueName: \"kubernetes.io/projected/5abe8e87-10f7-453b-872f-ca366598dd92-kube-api-access-7qwjr\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.497621 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.497605 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xhlv\" (UniqueName: \"kubernetes.io/projected/ee11341a-54bb-412f-ad81-2c8ef689247a-kube-api-access-2xhlv\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.498058 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.498035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dj8\" (UniqueName: \"kubernetes.io/projected/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-kube-api-access-v2dj8\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.498146 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.498126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.498897 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.498883 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmlw\" (UniqueName: \"kubernetes.io/projected/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-kube-api-access-4wmlw\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.517642 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.517625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:09.523975 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.523959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:09.531908 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.531887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:09.572665 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.571534 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:09.597185 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.593659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:09.599585 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.599528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:09.606470 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.606322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:09.612276 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.611947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:09.617952 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.617484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:09.638059 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.638024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:09.822507 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.822481 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bnjl2"] Oct 30 21:09:09.824983 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:09.824960 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9781bb5_7481_4803_8ef9_2eca6954c446.slice/crio-56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82 WatchSource:0}: Error finding container 56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82: Status 404 returned error can't find the container with id 56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82 Oct 30 21:09:09.833079 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.833050 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kmzkx"] Oct 30 21:09:09.841696 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.841671 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j9ct9"] Oct 30 21:09:09.879321 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.879276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:09.879603 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.879488 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:09.879603 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.879505 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65c6946-57jbj: secret "image-registry-tls" not found Oct 30 21:09:09.879603 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.879563 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls podName:dccf83b0-6149-4f2b-b3b1-db339a5a245e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.879545454 +0000 UTC m=+31.060465361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls") pod "image-registry-5b65c6946-57jbj" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e") : secret "image-registry-tls" not found Oct 30 21:09:09.887634 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.887607 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gh745"] Oct 30 21:09:09.890467 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:09.889153 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd6abf2_7709_4ef1_bed2_3f6b4b667bdf.slice/crio-b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a WatchSource:0}: Error finding container b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a: Status 404 returned error can't find the container with id b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a Oct 30 21:09:09.932292 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.932171 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc"] Oct 30 21:09:09.946614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.946594 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f"] Oct 30 21:09:09.948335 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:09.948309 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8386a1_1b52_440d_b134_9c42390f70ba.slice/crio-16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e WatchSource:0}: Error finding container 16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e: Status 404 returned error can't find the container with id 16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e Oct 30 21:09:09.948632 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:09.948616 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9931cd13_1775_4f3d_9300_d2cb16df7acf.slice/crio-a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e WatchSource:0}: Error finding container a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e: Status 404 returned error can't find the container with id a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e Oct 30 21:09:09.957966 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.955258 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch"] Oct 30 21:09:09.960204 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.960179 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w"] Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.985774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.985834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.985859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.985878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.985919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.985938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.985955 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.985997 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.985992644 +0000 UTC m=+31.166912536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.985998 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986026 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.985958 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986055 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5774b95-sq55b: secret "image-registry-tls" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986030 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.986020532 +0000 UTC m=+31.166940416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : configmap references non-existent config key: service-ca.crt Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:09.986099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986134 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:09.986986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986141 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls podName:7c5b111d-8074-4bda-8ac9-7d1f1f623c59 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.986131065 +0000 UTC m=+31.167050940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls") pod "dns-default-rp8hx" (UID: "7c5b111d-8074-4bda-8ac9-7d1f1f623c59") : secret "dns-default-metrics-tls" not found Oct 30 21:09:09.987936 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.986147333 +0000 UTC m=+31.167067208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : secret "router-metrics-certs-default" not found Oct 30 21:09:09.987936 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986163 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert podName:f71a2be6-1f45-43a6-ae97-8ebd007341fd nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.986157267 +0000 UTC m=+31.167077143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-phllg" (UID: "f71a2be6-1f45-43a6-ae97-8ebd007341fd") : secret "networking-console-plugin-cert" not found Oct 30 21:09:09.987936 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986172 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls podName:57293ead-fcc2-47e7-8033-d0beec008c51 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.986167815 +0000 UTC m=+31.167087691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls") pod "image-registry-66d5774b95-sq55b" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51") : secret "image-registry-tls" not found Oct 30 21:09:09.987936 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:09.986182 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls podName:5abe8e87-10f7-453b-872f-ca366598dd92 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:10.986177662 +0000 UTC m=+31.167097537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xr594" (UID: "5abe8e87-10f7-453b-872f-ca366598dd92") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:10.002581 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.002415 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5lcrb"] Oct 30 21:09:10.007908 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:10.007825 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee11341a_54bb_412f_ad81_2c8ef689247a.slice/crio-eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6 WatchSource:0}: Error finding container eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6: Status 404 returned error can't find the container with id eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6 Oct 30 21:09:10.008216 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.008132 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vws6h"] Oct 30 21:09:10.715353 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.715318 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" event={"ID":"2b8386a1-1b52-440d-b134-9c42390f70ba","Type":"ContainerStarted","Data":"16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e"} Oct 30 21:09:10.716818 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.716789 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" event={"ID":"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34","Type":"ContainerStarted","Data":"c24f04d8515b1cff4df3aa8f22a5dcd5abcdfc509065cc61fb3a35bd39c76d61"} Oct 30 21:09:10.718585 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.718518 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" event={"ID":"8a877cc0-db2f-42fc-9057-a62433401379","Type":"ContainerStarted","Data":"4039f00544feb1eecd65f3c8ea6dbd23196bb56d4eb107e99c4ffdc7aa87d2ad"} Oct 30 21:09:10.722962 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.722933 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" event={"ID":"9931cd13-1775-4f3d-9300-d2cb16df7acf","Type":"ContainerStarted","Data":"a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e"} Oct 30 21:09:10.738178 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.738150 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" event={"ID":"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe","Type":"ContainerStarted","Data":"a7dfee378ebf3084b8704440b079cfee52114d86cd86cf46621320b6426d4692"} Oct 30 21:09:10.738290 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.738229 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" event={"ID":"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe","Type":"ContainerStarted","Data":"d18517ffd425309b618710c74180cd5e1ec7919b9c61ac4043ad36f9ff3bb4b5"} Oct 30 21:09:10.753334 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.753089 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" event={"ID":"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75","Type":"ContainerStarted","Data":"d45a82b224817cd7f19b1aeb8e8542e56582d1ecc77dfe0a0d61ed0471d209a2"} Oct 30 21:09:10.753334 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.753129 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" event={"ID":"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75","Type":"ContainerStarted","Data":"ab2c2178540afbea79f7bf01d40d098687301535a6d26ee1eaf8227203bcfc00"} Oct 30 21:09:10.758579 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.758545 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" event={"ID":"d9781bb5-7481-4803-8ef9-2eca6954c446","Type":"ContainerStarted","Data":"ea1000181fb4a94a280e3fbadbde856f09d5808ce546700109e7449624846a5e"} Oct 30 21:09:10.758726 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.758709 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" event={"ID":"d9781bb5-7481-4803-8ef9-2eca6954c446","Type":"ContainerStarted","Data":"56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82"} Oct 30 21:09:10.761244 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.761227 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5lcrb" event={"ID":"ee11341a-54bb-412f-ad81-2c8ef689247a","Type":"ContainerStarted","Data":"eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6"} Oct 30 21:09:10.762997 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.762980 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vws6h" event={"ID":"08beba50-863a-4f30-a1ad-324cfde2159f","Type":"ContainerStarted","Data":"eae9a82fe50785ea85fb1db3de435afe0ae3e0fa2f94fbe0418a1485361b6fb9"} Oct 30 21:09:10.765890 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.765832 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" event={"ID":"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf","Type":"ContainerStarted","Data":"b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a"} Oct 30 21:09:10.775327 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.775290 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" podStartSLOduration=295.775280603 podStartE2EDuration="4m55.775280603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:10.774471084 +0000 UTC m=+30.955390983" watchObservedRunningTime="2026-10-30 21:09:10.775280603 +0000 UTC m=+30.956200500" Oct 30 21:09:10.833252 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.833020 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" podStartSLOduration=295.833002958 podStartE2EDuration="4m55.833002958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:10.830120022 +0000 UTC m=+31.011039919" watchObservedRunningTime="2026-10-30 21:09:10.833002958 +0000 UTC m=+31.013922858" Oct 30 21:09:10.898950 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.896383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:10.898950 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.897433 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:10.898950 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.897450 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65c6946-57jbj: secret "image-registry-tls" not found Oct 30 21:09:10.898950 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.897502 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls podName:dccf83b0-6149-4f2b-b3b1-db339a5a245e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:12.897484508 +0000 UTC m=+33.078404389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls") pod "image-registry-5b65c6946-57jbj" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e") : secret "image-registry-tls" not found Oct 30 21:09:10.999783 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.999703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:10.999783 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.999762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.999794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.999853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.999880 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.999940 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:10.999888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.999955 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:12.999936551 +0000 UTC m=+33.180856452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : secret "router-metrics-certs-default" not found Oct 30 21:09:10.999986 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:10.999943 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000007 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:12.999988042 +0000 UTC m=+33.180907928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:13.000015625 +0000 UTC m=+33.180935521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : configmap references non-existent config key: service-ca.crt Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000031 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000041 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert podName:f71a2be6-1f45-43a6-ae97-8ebd007341fd nodeName:}" failed. No retries permitted until 2026-10-30 21:09:13.000032472 +0000 UTC m=+33.180952354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-phllg" (UID: "f71a2be6-1f45-43a6-ae97-8ebd007341fd") : secret "networking-console-plugin-cert" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:11.000080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000128 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:11.000161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000169 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls podName:5abe8e87-10f7-453b-872f-ca366598dd92 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:13.000154667 +0000 UTC m=+33.181074577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xr594" (UID: "5abe8e87-10f7-453b-872f-ca366598dd92") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000196 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls podName:7c5b111d-8074-4bda-8ac9-7d1f1f623c59 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:13.000182731 +0000 UTC m=+33.181102636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls") pod "dns-default-rp8hx" (UID: "7c5b111d-8074-4bda-8ac9-7d1f1f623c59") : secret "dns-default-metrics-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000233 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000245 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5774b95-sq55b: secret "image-registry-tls" not found Oct 30 21:09:11.000309 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:11.000282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls podName:57293ead-fcc2-47e7-8033-d0beec008c51 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:13.000267453 +0000 UTC m=+33.181187363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls") pod "image-registry-66d5774b95-sq55b" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51") : secret "image-registry-tls" not found Oct 30 21:09:11.163171 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:11.163067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:11.358539 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:11.358362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:12.776916 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:12.776163 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" event={"ID":"2b8386a1-1b52-440d-b134-9c42390f70ba","Type":"ContainerStarted","Data":"75161bdd01cebf747b7db7fd31d44d011afba3ea9ac929c09e693b4a08ffd846"} Oct 30 21:09:12.818589 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:12.817870 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" podStartSLOduration=297.817849825 podStartE2EDuration="4m57.817849825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:10.873480759 +0000 UTC m=+31.054400659" watchObservedRunningTime="2026-10-30 21:09:12.817849825 +0000 UTC m=+32.998769733" Oct 30 21:09:12.922733 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:12.922691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:12.924334 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:12.923894 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:12.924334 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:12.923917 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65c6946-57jbj: secret "image-registry-tls" not found Oct 30 21:09:12.924334 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:12.923970 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls podName:dccf83b0-6149-4f2b-b3b1-db339a5a245e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:16.923952237 +0000 UTC m=+37.104872116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls") pod "image-registry-5b65c6946-57jbj" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e") : secret "image-registry-tls" not found Oct 30 21:09:12.961004 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:12.960975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024213 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024240 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5774b95-sq55b: secret "image-registry-tls" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024307 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls podName:57293ead-fcc2-47e7-8033-d0beec008c51 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.024287908 +0000 UTC m=+37.205207798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls") pod "image-registry-66d5774b95-sq55b" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51") : secret "image-registry-tls" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024311 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.024340648 +0000 UTC m=+37.205260540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024362 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024424 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.024409431 +0000 UTC m=+37.205329326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : secret "router-metrics-certs-default" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024483 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024526 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert podName:f71a2be6-1f45-43a6-ae97-8ebd007341fd nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.024512576 +0000 UTC m=+37.205432456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-phllg" (UID: "f71a2be6-1f45-43a6-ae97-8ebd007341fd") : secret "networking-console-plugin-cert" not found Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:13.025014 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024567 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:09:13.025878 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.024583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:13.025878 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024603 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls podName:7c5b111d-8074-4bda-8ac9-7d1f1f623c59 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.024591443 +0000 UTC m=+37.205511332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls") pod "dns-default-rp8hx" (UID: "7c5b111d-8074-4bda-8ac9-7d1f1f623c59") : secret "dns-default-metrics-tls" not found Oct 30 21:09:13.025878 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024643 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:13.025878 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024679 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls podName:5abe8e87-10f7-453b-872f-ca366598dd92 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.024668437 +0000 UTC m=+37.205588349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xr594" (UID: "5abe8e87-10f7-453b-872f-ca366598dd92") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:13.025878 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:13.024740 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.02472917 +0000 UTC m=+37.205649048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : configmap references non-existent config key: service-ca.crt Oct 30 21:09:13.097477 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.097337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:13.783520 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.783446 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" event={"ID":"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf","Type":"ContainerStarted","Data":"da2e4efe1ec3cc8826619b9e798022c013c17e6257bd77be1e07aeef4ee50cd1"} Oct 30 21:09:13.784332 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.784197 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:13.785435 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.785403 2576 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-gh745 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Oct 30 21:09:13.785557 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.785457 2576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" podUID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Oct 30 21:09:13.823011 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:13.822948 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" podStartSLOduration=231.623677093 podStartE2EDuration="3m53.822923656s" totalImagesPullingTime="2.199246563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:12.818966932 +0000 UTC m=+32.999886831" watchObservedRunningTime="2026-10-30 21:09:13.822923656 +0000 UTC m=+34.003843557" Oct 30 21:09:14.138159 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.138115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:14.138337 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.138212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:14.138480 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:14.138462 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:09:14.138557 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:14.138537 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:46.138516739 +0000 UTC m=+66.319436638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : secret "metrics-daemon-secret" not found Oct 30 21:09:14.143024 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.143002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:14.239560 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.239516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:14.242273 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.242091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:14.243038 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.242989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:14.533819 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.533735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:14.790761 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.790683 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vws6h" event={"ID":"08beba50-863a-4f30-a1ad-324cfde2159f","Type":"ContainerStarted","Data":"8762a741df60ec4b87cb543e44e5247c5256ab8f5180f5681ca879740c6a9d92"} Oct 30 21:09:14.792727 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.792710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/0.log" Oct 30 21:09:14.792825 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.792739 2576 generic.go:356] "Generic (PLEG): container finished" podID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" containerID="da2e4efe1ec3cc8826619b9e798022c013c17e6257bd77be1e07aeef4ee50cd1" exitCode=255 Oct 30 21:09:14.792871 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.792821 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" event={"ID":"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf","Type":"ContainerDied","Data":"da2e4efe1ec3cc8826619b9e798022c013c17e6257bd77be1e07aeef4ee50cd1"} Oct 30 21:09:14.793056 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.793041 2576 scope.go:122] "RemoveContainer" containerID="da2e4efe1ec3cc8826619b9e798022c013c17e6257bd77be1e07aeef4ee50cd1" Oct 30 21:09:14.795423 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.795359 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" event={"ID":"8a877cc0-db2f-42fc-9057-a62433401379","Type":"ContainerStarted","Data":"25f03af7906dbea31e9d84290487d8f67ca114773e579c8d47728e6266789f0e"} Oct 30 21:09:14.796984 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.796959 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" event={"ID":"9931cd13-1775-4f3d-9300-d2cb16df7acf","Type":"ContainerStarted","Data":"8b9ceacf795cd1da5eac60fdf96b134a26ab19fa78b2185efe057b3e73a1281e"} Oct 30 21:09:14.830613 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.830332 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" podStartSLOduration=232.048861973 podStartE2EDuration="3m55.830315544s" totalImagesPullingTime="3.781453571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:13.821635535 +0000 UTC m=+34.002555432" watchObservedRunningTime="2026-10-30 21:09:14.830315544 +0000 UTC m=+35.011235442" Oct 30 21:09:14.832204 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.831783 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-vws6h" podStartSLOduration=230.253451588 podStartE2EDuration="3m54.831770213s" totalImagesPullingTime="4.578318625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:14.828325631 +0000 UTC m=+35.009245530" watchObservedRunningTime="2026-10-30 21:09:14.831770213 +0000 UTC m=+35.012690112" Oct 30 21:09:14.893812 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.893784 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-85c8w"] Oct 30 21:09:14.900143 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.900115 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b9l4r"] Oct 30 21:09:14.902504 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:14.902477 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1e1f33_a5b9_442c_8589_970667a86890.slice/crio-ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894 WatchSource:0}: Error finding container ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894: Status 404 returned error can't find the container with id ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894 Oct 30 21:09:14.947690 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.942928 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" podStartSLOduration=239.300614749 podStartE2EDuration="4m3.942908353s" totalImagesPullingTime="4.642293604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:14.90458722 +0000 UTC m=+35.085507119" watchObservedRunningTime="2026-10-30 21:09:14.942908353 +0000 UTC m=+35.123828252" Oct 30 21:09:14.947690 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:14.943170 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" podStartSLOduration=239.33751835 podStartE2EDuration="4m3.943160732s" totalImagesPullingTime="4.605642382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:14.941691428 +0000 UTC m=+35.122611327" watchObservedRunningTime="2026-10-30 21:09:14.943160732 +0000 UTC m=+35.124080630" Oct 30 21:09:15.627443 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.627409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:15.635225 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.635202 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq"] Oct 30 21:09:15.675468 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.675446 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg"] Oct 30 21:09:15.675709 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.675691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:15.690543 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.690502 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:09:15.690633 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.690563 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:15.690736 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.690718 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kmcds\"" Oct 30 21:09:15.691859 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.691829 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq"] Oct 30 21:09:15.691964 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.691880 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg"] Oct 30 21:09:15.692034 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.692006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:15.771738 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.770551 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sq4d5"] Oct 30 21:09:15.774686 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.774630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:15.785791 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.785765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:15.792757 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.792687 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sq4d5"] Oct 30 21:09:15.803855 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.803831 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:09:15.803990 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.803886 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:09:15.803990 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.803947 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8rdf8\"" Oct 30 21:09:15.820023 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.819967 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" event={"ID":"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34","Type":"ContainerStarted","Data":"27b120b741aaa59f058b4d000cf36f70a98c897444e8999e82a61b0721a8bfc4"} Oct 30 21:09:15.835721 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.835694 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vb2td-csvl7"] Oct 30 21:09:15.849753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.849709 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-85c8w" event={"ID":"0c1e1f33-a5b9-442c-8589-970667a86890","Type":"ContainerStarted","Data":"871ed0d547f0f78d91b5374fdbc75dcfa70f178a34dca288926d44eb73e3d261"} Oct 30 21:09:15.849753 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.849743 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-85c8w" event={"ID":"0c1e1f33-a5b9-442c-8589-970667a86890","Type":"ContainerStarted","Data":"ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894"} Oct 30 21:09:15.850063 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.849764 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b9l4r" event={"ID":"4142b823-6b14-49f2-bf14-e8231f9296f3","Type":"ContainerStarted","Data":"241c8c8035a644c1d27ea3b08471378c6e7a34eca10748f69565a49feadabb37"} Oct 30 21:09:15.850063 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.849839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:15.850063 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.849957 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:15.850217 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.850188 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5lcrb" event={"ID":"ee11341a-54bb-412f-ad81-2c8ef689247a","Type":"ContainerStarted","Data":"7bd426045ea1772de111a5225669a18f37b76babc7174a701f31c34ca9bf27a2"} Oct 30 21:09:15.852500 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.852334 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:09:15.853083 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.853064 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/0.log" Oct 30 21:09:15.853177 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.853102 2576 generic.go:356] "Generic (PLEG): container finished" podID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" containerID="5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb" exitCode=255 Oct 30 21:09:15.853268 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.853246 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" event={"ID":"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf","Type":"ContainerDied","Data":"5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb"} Oct 30 21:09:15.853347 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.853291 2576 scope.go:122] "RemoveContainer" containerID="da2e4efe1ec3cc8826619b9e798022c013c17e6257bd77be1e07aeef4ee50cd1" Oct 30 21:09:15.854080 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.853661 2576 scope.go:122] "RemoveContainer" containerID="5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb" Oct 30 21:09:15.854080 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:15.853826 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gh745_openshift-console-operator(8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" podUID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" Oct 30 21:09:15.858193 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.858168 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vb2td-csvl7"] Oct 30 21:09:15.859871 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.859849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dn2f\" (UniqueName: \"kubernetes.io/projected/3a8e67f5-3ce1-412e-afb4-efbae8b2db98-kube-api-access-6dn2f\") pod \"migrator-66df5d496b-dv4lg\" (UID: \"3a8e67f5-3ce1-412e-afb4-efbae8b2db98\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:15.860030 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.859962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drzd\" (UniqueName: \"kubernetes.io/projected/9feefbd2-6be8-47ab-a427-e9e91ff48e1a-kube-api-access-6drzd\") pod \"migrator-66df5d496b-lxmgq\" (UID: \"9feefbd2-6be8-47ab-a427-e9e91ff48e1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:15.876160 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.876094 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" podStartSLOduration=239.231730563 podStartE2EDuration="4m4.87607887s" totalImagesPullingTime="5.644348307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:15.875332809 +0000 UTC m=+36.056252700" watchObservedRunningTime="2026-10-30 21:09:15.87607887 +0000 UTC m=+36.056998768" Oct 30 21:09:15.916614 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.916588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:15.950708 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.949413 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hh45f"] Oct 30 21:09:15.955285 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.955108 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5lcrb" podStartSLOduration=2.147586439 podStartE2EDuration="6.955091342s" totalImagesPullingTime="4.807504903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:15.953467344 +0000 UTC m=+36.134387244" watchObservedRunningTime="2026-10-30 21:09:15.955091342 +0000 UTC m=+36.136011240" Oct 30 21:09:15.960782 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.960494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:15.960782 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.960587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zwx\" (UniqueName: \"kubernetes.io/projected/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-api-access-g7zwx\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:15.960782 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.960645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:15.960782 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.960673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdf32f3-9268-4552-aa33-1b201380e88e-crio-socket\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:15.961070 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dn2f\" (UniqueName: \"kubernetes.io/projected/3a8e67f5-3ce1-412e-afb4-efbae8b2db98-kube-api-access-6dn2f\") pod \"migrator-66df5d496b-dv4lg\" (UID: \"3a8e67f5-3ce1-412e-afb4-efbae8b2db98\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:15.961130 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drzd\" (UniqueName: \"kubernetes.io/projected/9feefbd2-6be8-47ab-a427-e9e91ff48e1a-kube-api-access-6drzd\") pod \"migrator-66df5d496b-lxmgq\" (UID: \"9feefbd2-6be8-47ab-a427-e9e91ff48e1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:15.961248 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:15.961306 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:15.961360 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdf32f3-9268-4552-aa33-1b201380e88e-data-volume\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:15.961429 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:15.961511 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.961491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:15.982144 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.982118 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dn2f\" (UniqueName: \"kubernetes.io/projected/3a8e67f5-3ce1-412e-afb4-efbae8b2db98-kube-api-access-6dn2f\") pod \"migrator-66df5d496b-dv4lg\" (UID: \"3a8e67f5-3ce1-412e-afb4-efbae8b2db98\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:15.982284 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.982262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drzd\" (UniqueName: \"kubernetes.io/projected/9feefbd2-6be8-47ab-a427-e9e91ff48e1a-kube-api-access-6drzd\") pod \"migrator-66df5d496b-lxmgq\" (UID: \"9feefbd2-6be8-47ab-a427-e9e91ff48e1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:15.983481 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.983441 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hh45f"] Oct 30 21:09:15.983668 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:15.983546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.000494 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.000467 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:09:16.000638 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.000624 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:09:16.000851 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.000836 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tkfgn\"" Oct 30 21:09:16.001494 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.001475 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:09:16.001730 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.001718 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:09:16.002623 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.002586 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-85c8w" podStartSLOduration=34.286817085 podStartE2EDuration="35.002575059s" totalImagesPullingTime="715.757974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:16.001561141 +0000 UTC m=+36.182481038" watchObservedRunningTime="2026-10-30 21:09:16.002575059 +0000 UTC m=+36.183494955" Oct 30 21:09:16.011856 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.011697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:16.028840 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.028821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:16.062508 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.062618 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.062618 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdf32f3-9268-4552-aa33-1b201380e88e-data-volume\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.062618 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.062718 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.062718 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.062780 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zwx\" (UniqueName: \"kubernetes.io/projected/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-api-access-g7zwx\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.062814 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.062845 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdf32f3-9268-4552-aa33-1b201380e88e-crio-socket\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.062929 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.062914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdf32f3-9268-4552-aa33-1b201380e88e-crio-socket\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.063277 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.063259 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:16.063332 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.063322 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:16.563305388 +0000 UTC m=+36.744225267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:16.064567 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.064117 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.064567 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.064126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.064567 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.064196 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:09:16.064567 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.064240 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:16.564224702 +0000 UTC m=+36.745144581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : configmap "kube-rbac-proxy" not found Oct 30 21:09:16.064567 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.064340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.064567 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.064409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdf32f3-9268-4552-aa33-1b201380e88e-data-volume\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.085383 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.085334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.088887 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.088859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zwx\" (UniqueName: \"kubernetes.io/projected/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-api-access-g7zwx\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.163755 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.163527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/84eda726-2f51-4c10-a829-681fdbb559e2-signing-key\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.163755 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.163617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/84eda726-2f51-4c10-a829-681fdbb559e2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.163755 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.163640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rwq\" (UniqueName: \"kubernetes.io/projected/84eda726-2f51-4c10-a829-681fdbb559e2-kube-api-access-56rwq\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.183270 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.183241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:16.265159 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.265114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/84eda726-2f51-4c10-a829-681fdbb559e2-signing-key\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.265508 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.265377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/84eda726-2f51-4c10-a829-681fdbb559e2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.265508 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.265425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rwq\" (UniqueName: \"kubernetes.io/projected/84eda726-2f51-4c10-a829-681fdbb559e2-kube-api-access-56rwq\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.267809 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.267757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/84eda726-2f51-4c10-a829-681fdbb559e2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.284642 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.284590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/84eda726-2f51-4c10-a829-681fdbb559e2-signing-key\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.286409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.286370 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rwq\" (UniqueName: \"kubernetes.io/projected/84eda726-2f51-4c10-a829-681fdbb559e2-kube-api-access-56rwq\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.326365 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.326330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:16.330866 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.330841 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq"] Oct 30 21:09:16.333733 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:16.333701 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feefbd2_6be8_47ab_a427_e9e91ff48e1a.slice/crio-9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9 WatchSource:0}: Error finding container 9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9: Status 404 returned error can't find the container with id 9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9 Oct 30 21:09:16.349639 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.349608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:16.365695 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.365628 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg"] Oct 30 21:09:16.368454 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:16.368432 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8e67f5_3ce1_412e_afb4_efbae8b2db98.slice/crio-4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf WatchSource:0}: Error finding container 4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf: Status 404 returned error can't find the container with id 4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf Oct 30 21:09:16.494790 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.494760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:16.511206 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.511165 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vb2td-csvl7"] Oct 30 21:09:16.513496 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:16.513465 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd887faa8_91ab_4a8e_8b92_38a199ef7081.slice/crio-b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46 WatchSource:0}: Error finding container b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46: Status 404 returned error can't find the container with id b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46 Oct 30 21:09:16.570589 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.570561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.570702 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.570626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:16.570753 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.570734 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:16.570862 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.570847 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.570827549 +0000 UTC m=+37.751747427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:16.570921 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.570908 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:09:16.570977 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.570959 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:17.570945079 +0000 UTC m=+37.751864954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : configmap "kube-rbac-proxy" not found Oct 30 21:09:16.653334 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.653306 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hh45f"] Oct 30 21:09:16.657618 ip-10-0-133-149 kubenswrapper[2576]: W1030 21:09:16.657549 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84eda726_2f51_4c10_a829_681fdbb559e2.slice/crio-c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1 WatchSource:0}: Error finding container c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1: Status 404 returned error can't find the container with id c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1 Oct 30 21:09:16.864751 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.864718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:09:16.865200 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.865161 2576 scope.go:122] "RemoveContainer" containerID="5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb" Oct 30 21:09:16.865822 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.865362 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gh745_openshift-console-operator(8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" podUID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" Oct 30 21:09:16.870065 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.870037 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vb2td-csvl7" event={"ID":"d887faa8-91ab-4a8e-8b92-38a199ef7081","Type":"ContainerStarted","Data":"c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296"} Oct 30 21:09:16.870180 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.870087 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vb2td-csvl7" event={"ID":"d887faa8-91ab-4a8e-8b92-38a199ef7081","Type":"ContainerStarted","Data":"b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46"} Oct 30 21:09:16.873460 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.873438 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" event={"ID":"84eda726-2f51-4c10-a829-681fdbb559e2","Type":"ContainerStarted","Data":"71687814366c2556e576be65e571368907c2776dcc235359d28b47d326161b56"} Oct 30 21:09:16.873567 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.873475 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" event={"ID":"84eda726-2f51-4c10-a829-681fdbb559e2","Type":"ContainerStarted","Data":"c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1"} Oct 30 21:09:16.880185 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.880116 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" event={"ID":"9feefbd2-6be8-47ab-a427-e9e91ff48e1a","Type":"ContainerStarted","Data":"9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9"} Oct 30 21:09:16.883412 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.882955 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" event={"ID":"3a8e67f5-3ce1-412e-afb4-efbae8b2db98","Type":"ContainerStarted","Data":"4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf"} Oct 30 21:09:16.952551 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.952452 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" podStartSLOduration=1.952435227 podStartE2EDuration="1.952435227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:16.951028309 +0000 UTC m=+37.131948207" watchObservedRunningTime="2026-10-30 21:09:16.952435227 +0000 UTC m=+37.133355124" Oct 30 21:09:16.975534 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.975503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:16.977634 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.977608 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:16.977634 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.977633 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b65c6946-57jbj: secret "image-registry-tls" not found Oct 30 21:09:16.977804 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:16.977684 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls podName:dccf83b0-6149-4f2b-b3b1-db339a5a245e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:24.977666706 +0000 UTC m=+45.158586614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls") pod "image-registry-5b65c6946-57jbj" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e") : secret "image-registry-tls" not found Oct 30 21:09:16.992410 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:16.992333 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vb2td-csvl7" podStartSLOduration=1.9923146090000001 podStartE2EDuration="1.992314609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:16.990839463 +0000 UTC m=+37.171759361" watchObservedRunningTime="2026-10-30 21:09:16.992314609 +0000 UTC m=+37.173234516" Oct 30 21:09:17.077258 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:17.077452 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:17.077452 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:17.077452 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:17.077609 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:17.077609 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:17.077609 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.077554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:17.077725 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.077693 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:17.077769 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.077759 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls podName:5abe8e87-10f7-453b-872f-ca366598dd92 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.077740823 +0000 UTC m=+45.258660718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xr594" (UID: "5abe8e87-10f7-453b-872f-ca366598dd92") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:09:17.078173 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078153 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:09:17.078173 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078173 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5774b95-sq55b: secret "image-registry-tls" not found Oct 30 21:09:17.078328 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078219 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls podName:57293ead-fcc2-47e7-8033-d0beec008c51 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.078204869 +0000 UTC m=+45.259124744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls") pod "image-registry-66d5774b95-sq55b" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51") : secret "image-registry-tls" not found Oct 30 21:09:17.078573 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078557 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:09:17.078636 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078602 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.078588908 +0000 UTC m=+45.259508784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : secret "router-metrics-certs-default" not found Oct 30 21:09:17.078707 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078672 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:17.078769 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078709 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.078697459 +0000 UTC m=+45.259617347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:17.078769 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078761 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:09:17.078865 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert podName:f71a2be6-1f45-43a6-ae97-8ebd007341fd nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.078783162 +0000 UTC m=+45.259703038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-phllg" (UID: "f71a2be6-1f45-43a6-ae97-8ebd007341fd") : secret "networking-console-plugin-cert" not found Oct 30 21:09:17.078865 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078843 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:09:17.078963 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078871 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls podName:7c5b111d-8074-4bda-8ac9-7d1f1f623c59 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.078862105 +0000 UTC m=+45.259781984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls") pod "dns-default-rp8hx" (UID: "7c5b111d-8074-4bda-8ac9-7d1f1f623c59") : secret "dns-default-metrics-tls" not found Oct 30 21:09:17.078963 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.078930 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.078920569 +0000 UTC m=+45.259840458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : configmap references non-existent config key: service-ca.crt Oct 30 21:09:17.297635 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.297547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:17.592811 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.585513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:17.592811 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.585595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:17.592811 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.585684 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:09:17.592811 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.585733 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:19.58572019 +0000 UTC m=+39.766640066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : configmap "kube-rbac-proxy" not found Oct 30 21:09:17.592811 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.586155 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:17.592811 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:17.586215 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:19.586200285 +0000 UTC m=+39.767120174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:17.734030 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.733992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:17.935796 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:17.935760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.085713 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.085669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.336499 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.336461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.476435 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.476328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.893069 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.893028 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" event={"ID":"3a8e67f5-3ce1-412e-afb4-efbae8b2db98","Type":"ContainerStarted","Data":"175ebc2bea164e614dcfee0d139331a188c2d99e721c0d5eaeb6e66ca33a5ba8"} Oct 30 21:09:18.894871 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.894838 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" event={"ID":"9feefbd2-6be8-47ab-a427-e9e91ff48e1a","Type":"ContainerStarted","Data":"c72adabbfd379150e5fa27c733a361b3af10eed3360d0d8b44331ab257196720"} Oct 30 21:09:18.936170 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.936123 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" podStartSLOduration=2.079791722 podStartE2EDuration="3.936102026s" totalImagesPullingTime="1.856310304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:18.934440272 +0000 UTC m=+39.115360170" watchObservedRunningTime="2026-10-30 21:09:18.936102026 +0000 UTC m=+39.117021922" Oct 30 21:09:18.938064 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.938037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.971313 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:18.971262 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" podStartSLOduration=2.083872099 podStartE2EDuration="3.971245821s" totalImagesPullingTime="1.887373722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:18.970255774 +0000 UTC m=+39.151175684" watchObservedRunningTime="2026-10-30 21:09:18.971245821 +0000 UTC m=+39.152165719" Oct 30 21:09:19.067671 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.067642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:19.402522 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.402482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:19.572285 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.572249 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:19.572756 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.572737 2576 scope.go:122] "RemoveContainer" containerID="5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb" Oct 30 21:09:19.572975 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:19.572952 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gh745_openshift-console-operator(8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" podUID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" Oct 30 21:09:19.600148 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.600120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:19.604312 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.604288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:19.604456 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:19.604433 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:09:19.604510 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.604483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:19.604510 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:19.604496 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.604480214 +0000 UTC m=+43.785400113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : configmap "kube-rbac-proxy" not found Oct 30 21:09:19.604613 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:19.604583 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:19.604664 ip-10-0-133-149 kubenswrapper[2576]: E1030 21:09:19.604649 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.604618218 +0000 UTC m=+43.785538096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:19.631245 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.631216 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c7kgv_9bafe23a-4bb7-477d-81a1-6024f5c3d9d4/dns-node-resolver/0.log" Oct 30 21:09:19.800889 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:19.800804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:20.005778 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:20.005730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:20.147409 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:20.147372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:20.290872 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:20.290838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:20.431917 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:20.431866 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-87n87_22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e/node-ca/0.log" Oct 30 21:09:20.662647 ip-10-0-133-149 kubenswrapper[2576]: I1030 21:09:20.662608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:20.738151 ip-10-0-133-149 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:09:20.749824 ip-10-0-133-149 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:09:20.749987 ip-10-0-133-149 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:09:20.750160 ip-10-0-133-149 systemd[1]: kubelet.service: Consumed 3.204s CPU time. Oct 30 21:09:20.760041 ip-10-0-133-149 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:09:20.834688 ip-10-0-133-149 kubenswrapper[5039]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:09:20.834688 ip-10-0-133-149 kubenswrapper[5039]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:09:20.834688 ip-10-0-133-149 kubenswrapper[5039]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:09:20.834688 ip-10-0-133-149 kubenswrapper[5039]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:09:20.836750 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836667 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.836750 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836749 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.836750 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836753 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836757 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836760 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836763 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836766 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836769 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836772 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836774 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836777 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836780 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836782 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836785 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836788 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836791 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836794 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836796 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836799 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836802 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836804 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836807 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.836861 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836809 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836812 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836815 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836818 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836826 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836830 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836832 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836835 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836837 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836840 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836843 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836845 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836848 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836851 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836854 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836856 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836859 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836862 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836864 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836867 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.837363 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836870 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836872 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836875 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836878 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836882 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836888 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836891 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836894 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836897 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836900 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836902 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836905 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836908 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836911 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836914 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836916 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836919 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836925 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836928 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.837873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836931 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836933 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836936 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836938 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836941 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836944 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836947 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836949 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836952 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836954 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836956 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836959 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836961 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836964 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836966 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836969 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836972 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836974 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836977 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.838343 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836979 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836982 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836984 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836987 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836990 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836993 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836996 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.836998 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837001 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837003 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837006 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837008 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837012 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837015 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837017 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837020 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837024 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837027 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837030 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837033 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.838830 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837035 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837038 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837041 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837043 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837046 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837049 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837800 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837807 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837810 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837813 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837816 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837819 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837822 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837825 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837829 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837833 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837836 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837839 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837841 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837843 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.839348 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837846 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837849 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837852 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837854 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837857 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837862 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837866 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837868 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837871 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837873 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837875 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837878 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837881 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837883 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837886 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837889 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837892 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837894 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837897 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.839874 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837899 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837902 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837904 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837907 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837910 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837913 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837915 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837918 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837920 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837923 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837925 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837927 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837930 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837932 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837935 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837937 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837940 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837943 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837945 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837951 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.840381 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837954 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837957 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837959 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837962 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837964 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837967 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837969 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837973 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837977 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837980 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837983 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837985 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837988 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837990 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837993 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837995 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.837998 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838000 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838003 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.840894 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838006 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838008 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838011 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838013 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838016 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838018 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838020 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838023 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838026 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838028 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838031 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838033 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838035 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838039 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838041 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838044 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838046 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838049 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838051 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838054 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.841353 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838057 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838059 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838062 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838064 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838066 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838069 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838071 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838074 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838077 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838079 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838082 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838087 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838090 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838092 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838608 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838615 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838618 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838621 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838624 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838627 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.841893 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838629 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838632 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838636 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838640 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838643 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838647 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838649 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838652 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838655 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838657 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838660 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838663 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838665 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838668 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838670 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838673 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838675 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838678 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838680 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.842433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838683 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838685 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838688 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838690 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838695 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838698 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838700 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838703 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838705 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838708 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838710 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838713 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838715 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838717 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838720 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838723 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838725 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838728 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838731 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838734 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.842892 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838736 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838739 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838741 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838744 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838746 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838749 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838751 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838754 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838756 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838759 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838761 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838764 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838766 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838769 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838772 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838774 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838778 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838781 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838784 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838786 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.843377 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838789 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838791 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838793 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838796 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838798 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838801 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838803 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838806 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838809 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838811 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838814 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838816 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838819 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838821 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838824 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838826 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838829 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838831 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838834 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838837 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.843903 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838840 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838843 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838846 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838848 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838850 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838853 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838856 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838858 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838862 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838865 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838867 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838870 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838873 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838877 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838880 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838885 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838888 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838890 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838893 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838895 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.844410 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.838898 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.838986 5039 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.838995 5039 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839004 5039 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839011 5039 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839018 5039 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839022 5039 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839027 5039 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839031 5039 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839034 5039 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839037 5039 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839041 5039 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839044 5039 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839047 5039 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839050 5039 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839053 5039 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839056 5039 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839058 5039 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839061 5039 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839066 5039 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839070 5039 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839073 5039 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839076 5039 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839080 5039 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:09:20.845280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839084 5039 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839087 5039 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839090 5039 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839098 5039 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839101 5039 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839104 5039 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839107 5039 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839110 5039 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839113 5039 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839118 5039 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839121 5039 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839124 5039 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839126 5039 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839130 5039 flags.go:64] FLAG: --enable-server="true" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839133 5039 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839137 5039 flags.go:64] FLAG: --event-burst="100" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839141 5039 flags.go:64] FLAG: --event-qps="50" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839144 5039 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839147 5039 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839150 5039 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839154 5039 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839156 5039 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839160 5039 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839163 5039 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839166 5039 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:09:20.846043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839169 5039 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839172 5039 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839174 5039 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839180 5039 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839183 5039 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839186 5039 flags.go:64] FLAG: --feature-gates="" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839190 5039 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839193 5039 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839196 5039 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839199 5039 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839204 5039 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839207 5039 flags.go:64] FLAG: --help="false" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839210 5039 flags.go:64] FLAG: --hostname-override="ip-10-0-133-149.ec2.internal" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839213 5039 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839216 5039 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839219 5039 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839222 5039 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839225 5039 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839228 5039 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839231 5039 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839234 5039 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839237 5039 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839240 5039 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839243 5039 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:09:20.846683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839246 5039 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839250 5039 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839252 5039 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839255 5039 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839258 5039 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839261 5039 flags.go:64] FLAG: --lock-file="" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839264 5039 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839266 5039 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839269 5039 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839275 5039 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839278 5039 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839283 5039 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839286 5039 flags.go:64] FLAG: --logging-format="text" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839289 5039 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839292 5039 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839295 5039 flags.go:64] FLAG: --manifest-url="" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839298 5039 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839302 5039 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839307 5039 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839310 5039 flags.go:64] FLAG: --max-pods="110" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839313 5039 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839316 5039 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839319 5039 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839322 5039 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839325 5039 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:09:20.847304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839328 5039 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839331 5039 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839338 5039 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839341 5039 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839344 5039 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839347 5039 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839350 5039 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839353 5039 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839356 5039 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839359 5039 flags.go:64] FLAG: --port="10250" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839362 5039 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839370 5039 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-041c6ccd130168185" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839373 5039 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839376 5039 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839379 5039 flags.go:64] FLAG: --register-node="true" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839382 5039 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839385 5039 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839401 5039 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839408 5039 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839411 5039 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839415 5039 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839418 5039 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839421 5039 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839424 5039 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839427 5039 flags.go:64] FLAG: --runonce="false" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839431 5039 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:09:20.848072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839435 5039 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839438 5039 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839441 5039 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839444 5039 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839447 5039 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839450 5039 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839453 5039 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839456 5039 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839459 5039 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839462 5039 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839465 5039 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839468 5039 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839470 5039 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839474 5039 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839479 5039 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839482 5039 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839485 5039 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839489 5039 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839492 5039 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839495 5039 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839498 5039 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839501 5039 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839504 5039 flags.go:64] FLAG: --v="2" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839508 5039 flags.go:64] FLAG: --version="false" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839514 5039 flags.go:64] FLAG: --vmodule="" Oct 30 21:09:20.848725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839517 5039 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839521 5039 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839657 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839661 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839664 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839666 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839670 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839673 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839675 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839678 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839681 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839683 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839685 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839688 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839690 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839693 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839695 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839698 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839701 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839703 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.849550 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839706 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839709 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839711 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839714 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839716 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839719 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839722 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839724 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839727 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839729 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839732 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839735 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839738 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839741 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839749 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839753 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839755 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839758 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839762 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.850147 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839765 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839767 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839770 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839772 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839775 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839778 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839780 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839782 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839785 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839787 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839790 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839793 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839795 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839798 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839800 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839803 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839806 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839809 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839811 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839814 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.850691 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839816 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839819 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839821 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839824 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839828 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839831 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839833 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839836 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839840 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839844 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839846 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839850 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839853 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839856 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839858 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839860 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839863 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839865 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839868 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839871 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.851185 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839873 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839876 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839878 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839881 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839884 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839887 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839889 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839892 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839895 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839899 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839903 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839906 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839909 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839912 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839915 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839917 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839921 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839924 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839927 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.851681 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839930 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839933 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839935 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839938 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839943 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839945 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839948 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839951 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839953 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.839956 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.839981 5039 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840022 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840027 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840030 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840033 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.852144 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840036 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840038 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840041 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840043 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840046 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840048 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840051 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840054 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840057 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840059 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840062 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840064 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840067 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840069 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840072 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840077 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840079 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840082 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840084 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.852670 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840087 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840089 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840093 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840096 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840098 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840101 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840104 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840106 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840108 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840111 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840113 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840116 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840118 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840121 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840124 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840126 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840129 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840131 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840134 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840136 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.853220 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840139 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840142 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840144 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840147 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840149 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840152 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840154 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840157 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840161 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840164 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840166 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840169 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840171 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840174 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840178 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840181 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840183 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840186 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840189 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.853765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840191 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840194 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840196 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840199 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840201 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840203 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840206 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840209 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840211 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840214 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840216 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840219 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840221 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840224 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840226 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840229 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840232 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840234 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840237 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.854219 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840239 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840242 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840246 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840249 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840253 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840257 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840260 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840263 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840267 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840271 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840273 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840276 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840278 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840281 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840284 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840286 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840289 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840291 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840294 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840296 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.854714 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840299 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840301 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840304 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840306 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.840309 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.843809 5039 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.843825 5039 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.843842 5039 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.843847 5039 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843904 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843909 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843913 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843916 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843919 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843922 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843924 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843927 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843929 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843932 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.855210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843936 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843940 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843943 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843946 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843949 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843951 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843954 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843956 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843959 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843962 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843965 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843968 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843970 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843973 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843976 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843978 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843981 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843983 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843986 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843989 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.855873 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843992 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843994 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.843997 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844000 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844002 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844005 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844007 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844010 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844012 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844015 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844017 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844020 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844022 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844025 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844028 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844030 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844033 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844036 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844038 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844041 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.856337 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844044 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844046 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844049 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844051 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844054 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844057 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844059 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844062 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844064 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844067 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844069 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844072 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844074 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844077 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844079 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844082 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844085 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844087 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844090 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.856821 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844092 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844095 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844097 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844100 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844102 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844104 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844107 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844109 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844112 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844115 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844118 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844120 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844123 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844125 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844128 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844130 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844133 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844136 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844138 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.857358 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844141 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844143 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844146 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844148 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844151 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844153 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844156 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844158 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844161 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844165 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844169 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844172 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844175 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844178 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844180 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844183 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844186 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.857828 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844188 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.844215 5039 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.844641 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845002 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845006 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845012 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845016 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845021 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845027 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845031 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845036 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845041 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845045 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845056 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845061 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845065 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.858234 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845069 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845073 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845079 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845087 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845092 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845097 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845101 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845105 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845109 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845118 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845124 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845130 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845136 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845141 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845146 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845150 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845155 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845159 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845163 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.858749 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845167 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845172 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845176 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845185 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845189 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845193 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845198 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845202 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845206 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845211 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845215 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845219 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845223 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845227 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845232 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845236 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845245 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845249 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845254 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845258 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.859270 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845262 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845267 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845271 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845275 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845280 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845284 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845289 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845293 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845303 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845307 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845311 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845316 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845320 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845324 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845328 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845332 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845336 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845341 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845345 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845349 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.859826 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845353 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845363 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845367 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845371 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845375 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845380 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845421 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845455 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845462 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845468 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845473 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845479 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845484 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845489 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845494 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845500 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845505 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845511 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845516 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.860376 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845527 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845531 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845537 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845542 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845548 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845553 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845558 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845563 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845568 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845579 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845585 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845592 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845602 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845608 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.845825 5039 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845878 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:09:20.861199 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845884 5039 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845887 5039 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845891 5039 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845894 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845896 5039 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845899 5039 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845902 5039 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845904 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845907 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845909 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845912 5039 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845915 5039 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845919 5039 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845921 5039 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845924 5039 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845928 5039 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845931 5039 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845933 5039 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845936 5039 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845938 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:09:20.861838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845941 5039 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845944 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845947 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845949 5039 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845952 5039 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845954 5039 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845960 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845963 5039 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845967 5039 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845969 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845972 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845975 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845978 5039 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845980 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845983 5039 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845986 5039 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845989 5039 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845991 5039 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845993 5039 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845996 5039 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:09:20.862686 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.845999 5039 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846003 5039 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846006 5039 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846008 5039 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846011 5039 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846013 5039 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846016 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846019 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846021 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846024 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846026 5039 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846029 5039 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846031 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846034 5039 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846036 5039 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846039 5039 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846043 5039 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846046 5039 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846050 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846052 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:09:20.863208 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846056 5039 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846058 5039 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846061 5039 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846065 5039 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846069 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846072 5039 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846075 5039 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846078 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846080 5039 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846083 5039 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846085 5039 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846088 5039 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846092 5039 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846096 5039 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846100 5039 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846103 5039 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846106 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846108 5039 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846111 5039 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:09:20.863740 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846114 5039 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846116 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846119 5039 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846121 5039 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846124 5039 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846126 5039 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846129 5039 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846131 5039 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846134 5039 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846137 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846139 5039 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846142 5039 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846145 5039 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846148 5039 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846150 5039 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846153 5039 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846155 5039 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846158 5039 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846160 5039 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:09:20.864263 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846162 5039 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846165 5039 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846167 5039 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846171 5039 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846174 5039 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846176 5039 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:20.846179 5039 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.846244 5039 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847208 5039 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847262 5039 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847448 5039 server.go:1014] "Starting client certificate rotation" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847593 5039 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847649 5039 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:03:41 +0000 UTC" deadline="2028-05-09 02:12:47.511052196 +0000 UTC" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847688 5039 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13349h3m26.66336737s" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.847997 5039 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.849001 5039 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.850187 5039 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:09:20.865048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.851780 5039 log.go:25] "Validated CRI v1 image API" Oct 30 21:09:20.865522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.852492 5039 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:09:20.865522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.854490 5039 fs.go:135] Filesystem UUIDs: map[6289a759-81b1-426e-bec7-40134eecedc5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bc089313-3d17-4db7-bc12-f8c2c2c6a83f:/dev/nvme0n1p4] Oct 30 21:09:20.865621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.854504 5039 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/241c8c8035a644c1d27ea3b08471378c6e7a34eca10748f69565a49feadabb37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/241c8c8035a644c1d27ea3b08471378c6e7a34eca10748f69565a49feadabb37/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4039f00544feb1eecd65f3c8ea6dbd23196bb56d4eb107e99c4ffdc7aa87d2ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4039f00544feb1eecd65f3c8ea6dbd23196bb56d4eb107e99c4ffdc7aa87d2ad/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d900c012506d9ce2649bfe5419204d063aafe5f25b90b291508da8e18db8a74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d900c012506d9ce2649bfe5419204d063aafe5f25b90b291508da8e18db8a74/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b3525026d116424ba68c89e44306b992182460015bf1de7199480c9bde41f6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b3525026d116424ba68c89e44306b992182460015bf1de7199480c9bde41f6c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab2c2178540afbea79f7bf01d40d098687301535a6d26ee1eaf8227203bcfc00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab2c2178540afbea79f7bf01d40d098687301535a6d26ee1eaf8227203bcfc00/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c24f04d8515b1cff4df3aa8f22a5dcd5abcdfc509065cc61fb3a35bd39c76d61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c24f04d8515b1cff4df3aa8f22a5dcd5abcdfc509065cc61fb3a35bd39c76d61/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d18517ffd425309b618710c74180cd5e1ec7919b9c61ac4043ad36f9ff3bb4b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d18517ffd425309b618710c74180cd5e1ec7919b9c61ac4043ad36f9ff3bb4b5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7d9141568c42fde15f5e64a81efffa2b58b10ce4ef5d4a0213878a91069b1c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7d9141568c42fde15f5e64a81efffa2b58b10ce4ef5d4a0213878a91069b1c5/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eae9a82fe50785ea85fb1db3de435afe0ae3e0fa2f94fbe0418a1485361b6fb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eae9a82fe50785ea85fb1db3de435afe0ae3e0fa2f94fbe0418a1485361b6fb9/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08beba50-863a-4f30-a1ad-324cfde2159f/volumes/kubernetes.io~projected/kube-api-access-bwjnp:{mountpoint:/var/lib/kubelet/pods/08beba50-863a-4f30-a1ad-324cfde2159f/volumes/kubernetes.io~projected/kube-api-access-bwjnp major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08beba50-863a-4f30-a1ad-324cfde2159f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/08beba50-863a-4f30-a1ad-324cfde2159f/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1e1f33-a5b9-442c-8589-970667a86890/volumes/kubernetes.io~projected/kube-api-access-ncd4q:{mountpoint:/var/lib/kubelet/pods/0c1e1f33-a5b9-442c-8589-970667a86890/volumes/kubernetes.io~projected/kube-api-access-ncd4q major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/106e7cc6-df7c-4e5a-8876-1c0456a35023/volumes/kubernetes.io~projected/kube-api-access-jw94c:{mountpoint:/var/lib/kubelet/pods/106e7cc6-df7c-4e5a-8876-1c0456a35023/volumes/kubernetes.io~projected/kube-api-access-jw94c major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe/volumes/kubernetes.io~projected/kube-api-access-c4dgl:{mountpoint:/var/lib/kubelet/pods/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe/volumes/kubernetes.io~projected/kube-api-access-c4dgl major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/209912e2-5544-4a98-9282-ed7e28c449d0/volumes/kubernetes.io~projected/kube-api-access-v4cnl:{mountpoint:/var/lib/kubelet/pods/209912e2-5544-4a98-9282-ed7e28c449d0/volumes/kubernetes.io~projected/kube-api-access-v4cnl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21966c11-af60-40d2-bde4-d6cb8296cad4/volumes/kubernetes.io~projected/kube-api-access-tgbcm:{mountpoint:/var/lib/kubelet/pods/21966c11-af60-40d2-bde4-d6cb8296cad4/volumes/kubernetes.io~projected/kube-api-access-tgbcm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e/volumes/kubernetes.io~projected/kube-api-access-bmlpj:{mountpoint:/var/lib/kubelet/pods/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e/volumes/kubernetes.io~projected/kube-api-access-bmlpj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2459def9-9723-4de1-9a93-5a2dfb7a3cca/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2459def9-9723-4de1-9a93-5a2dfb7a3cca/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b8386a1-1b52-440d-b134-9c42390f70ba/volumes/kubernetes.io~projected/kube-api-access-5jfvg:{mountpoint:/var/lib/kubelet/pods/2b8386a1-1b52-440d-b134-9c42390f70ba/volumes/kubernetes.io~projected/kube-api-access-5jfvg major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a8e67f5-3ce1-412e-afb4-efbae8b2db98/volumes/kubernetes.io~projected/kube-api-access-6dn2f:{mountpoint:/var/lib/kubelet/pods/3a8e67f5-3ce1-412e-afb4-efbae8b2db98/volumes/kubernetes.io~projected/kube-api-access-6dn2f major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4142b823-6b14-49f2-bf14-e8231f9296f3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4142b823-6b14-49f2-bf14-e8231f9296f3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c/volumes/kubernetes.io~projected/kube-api-access-l656s:{mountpoint:/var/lib/kubelet/pods/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c/volumes/kubernetes.io~projected/kube-api-access-l656s major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34/volumes/kubernetes.io~projected/kube-api-access-wmf2d:{mountpoint:/var/lib/kubelet/pods/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34/volumes/kubernetes.io~projected/kube-api-access-wmf2d major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~projected/kube-api-access-bsgr4:{mountpoint:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~projected/kube-api-access-bsgr4 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abe8e87-10f7-453b-872f-ca366598dd92/volumes/kubernetes.io~projected/kube-api-access-7qwjr:{mountpoint:/var/lib/kubelet/pods/5abe8e87-10f7-453b-872f-ca366598dd92/volumes/kubernetes.io~projected/kube-api-access-7qwjr major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5b111d-8074-4bda-8ac9-7d1f1f623c59/volumes/kubernetes.io~projected/kube-api-access-4wmlw:{mountpoint:/var/lib/kubelet/pods/7c5b111d-8074-4bda-8ac9-7d1f1f623c59/volumes/kubernetes.io~projected/kube-api-access-4wmlw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~projected/kube-api-access-q4zh2:{mountpoint:/var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~projected/kube-api-access-q4zh2 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~secret/default-certificate major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84eda726-2f51-4c10-a829-681fdbb559e2/volumes/kubernetes.io~projected/kube-api-access-56rwq:{mountpoint:/var/lib/kubelet/pods/84eda726-2f51-4c10-a829-681fdbb559e2/volumes/kubernetes.io~projected/kube-api-access-56rwq major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84eda726-2f51-4c10-a829-681fdbb559e2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/84eda726-2f51-4c10-a829-681fdbb559e2/volumes/kubernetes.io~secret/signing-key major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a877cc0-db2f-42fc-9057-a62433401379/volumes/kubernetes.io~projected/kube-api-access-7x7bn:{mountpoint:/var/lib/kubelet/pods/8a877cc0-db2f-42fc-9057-a62433401379/volumes/kubernetes.io~projected/kube-api-access-7x7bn major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a877cc0-db2f-42fc-9057-a62433401379/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8a877cc0-db2f-42fc-9057-a62433401379/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/volumes/kubernetes.io~projected/kube-api-access-gm8rm:{mountpoint:/var/lib/kubelet/pods/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/volumes/kubernetes.io~projected/kube-api-access-gm8rm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dba424-9ecd-43b9-a09e-8c643019e969/volumes/kubernetes.io~projected/kube-api-access-gcnsn:{mountpoint:/var/lib/kubelet/pods/95dba424-9ecd-43b9-a09e-8c643019e969/volumes/kubernetes.io~projected/kube-api-access-gcnsn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9931cd13-1775-4f3d-9300-d2cb16df7acf/volumes/kubernetes.io~projected/kube-api-access-mwg9v:{mountpoint:/var/lib/kubelet/pods/9931cd13-1775-4f3d-9300-d2cb16df7acf/volumes/kubernetes.io~projected/kube-api-access-mwg9v major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9931cd13-1775-4f3d-9300-d2cb16df7acf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9931cd13-1775-4f3d-9300-d2cb16df7acf/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4/volumes/kubernetes.io~projected/kube-api-access-bvkt5:{mountpoint:/var/lib/kubelet/pods/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4/volumes/kubernetes.io~projected/kube-api-access-bvkt5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9feefbd2-6be8-47ab-a427-e9e91ff48e1a/volumes/kubernetes.io~projected/kube-api-access-6drzd:{mountpoint:/var/lib/kubelet/pods/9feefbd2-6be8-47ab-a427-e9e91ff48e1a/volumes/kubernetes.io~projected/kube-api-access-6drzd major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~projected/kube-api-access-l2f2m:{mountpoint:/var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~projected/kube-api-access-l2f2m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volumes/kubernetes.io~projected/kube-api-access-tqbks:{mountpoint:/var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volumes/kubernetes.io~projected/kube-api-access-tqbks major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75/volumes/kubernetes.io~projected/kube-api-access-zwc6h:{mountpoint:/var/lib/kubelet/pods/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75/volumes/kubernetes.io~projected/kube-api-access-zwc6h major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d887faa8-91ab-4a8e-8b92-38a199ef7081/volumes/kubernetes.io~projected/kube-api-access-6zhcn:{mountpoint:/var/lib/kubelet/pods/d887faa8-91ab-4a8e-8b92-38a199ef7081/volumes/kubernetes.io~projected/kube-api-access-6zhcn major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9781bb5-7481-4803-8ef9-2eca6954c446/volumes/kubernetes.io~projected/kube-api-access-bkscn:{mountpoint:/var/lib/kubelet/pods/d9781bb5-7481-4803-8ef9-2eca6954c446/volumes/kubernetes.io~projected/kube-api-access-bkscn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5/volumes/kubernetes.io~projected/kube-api-access-v2dj8:{mountpoint:/var/lib/kubelet/pods/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5/volumes/kubernetes.io~projected/kube-api-access-v2dj8 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~projected/kube-api-access-jwk4q:{mountpoint:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~projected/kube-api-access-jwk4q major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee11341a-54bb-412f-ad81-2c8ef689247a/volumes/kubernetes.io~projected/kube-api-access-2xhlv:{mountpoint:/var/lib/kubelet/pods/ee11341a-54bb-412f-ad81-2c8ef689247a/volumes/kubernetes.io~projected/kube-api-access-2xhlv major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee11341a-54bb-412f-ad81-2c8ef689247a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ee11341a-54bb-412f-ad81-2c8ef689247a/volumes/kubernetes.io~secret/cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffdf32f3-9268-4552-aa33-1b201380e88e/volumes/kubernetes.io~projected/kube-api-access-g7zwx:{mountpoint:/var/lib/kubelet/pods/ffdf32f3-9268-4552-aa33-1b201380e88e/volumes/kubernetes.io~projected/kube-api-access-g7zwx major:0 minor:345 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/81a6e83c3fe3cbad357f3d3473b2849be9956c0473c91c5554aca1c342ed16cc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/597cee4129ae2b1b02aa3c99b59514219ff8248000101d1a8ea99c1abdb5515a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/df872925d03aaa378d707bb737f0ac03727f024699b9fc4d58253d01a157a102/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/43787200fa1637390136b02496a02d2c6fdc78b5e022076f45f5e2171ee16bd3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8d7de478a806882a30ca8537d45576ebda5a56ef07a7a0c87b47a8ec0f2c7598/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f1262d626386deab9891a974dd77807eb7f13ff4b4a27724d059d0269d6c07f9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b41833575aed11c5604ecb82821fde6819e5e9383a44124cc44d89989dfbbd21/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f91ec6ac22778b0021997bb141796b4a1ed47f1d32dc1d01250772eee95dd7fb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/55dae789ccc18d50586d92affb1b60cb54b7fbb4236bb90e135615742c8528af/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c0c46a75db64c059d46b43d21d36d63697673e20e2c8648f8669a61b3e0e0f14/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/cb942adf3ffe2fb4704ade1e62cb74a1eea236eb444f37bcf6d060b66867cf29/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/07536d29a1ac5e5493316f182685ca296254a7504ebf81954ae47bc6afbe8192/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/be4fd6a915ff7bb34396656becd74ad3046e7ee86f97d7c71380720db0a8e9fc/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b0df325f7e4923de39bcc1cf78cd5685c620fb0cb9a1aedee60ac7b9a85fe076/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f14e3779e3230ab19780c14d0c115f3aa2e779d03a897cbb6c84b5c6f167212d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/191546c91a5e95413bc17ba5cfe4417e714ab0a91d345b88cb30972086373078/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/05a570d382d1379dd14b278091d73012f5d2abb2ca6e6edc5d2565dfed8a42f1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/165dfc03ba1b7fe33629a29e35f5605a66140514758274fb5b49934633bf022c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/83aa1b37d6d05828509bd45b82707dfbc60f0b2a51aa04d873ea6bd6c479c0b9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/90c68f54851ba1040680c4a73de595e16ef49f640aca66fff5b0434b1c9c128a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3fc75554111d2a02be407444bde0dfcf5df889545275518498e6bf572fa606e5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4c19603220f8127bc479b5f42b9dfc9fd8e2bde865af776c93be39366db92a02/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fd28c1282c992fd7d83585d38418d97aa0d0cd35d07fdc6cfa74473e31d03369/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/41a1a54d9c3f711120e39fc57d7607eee1ca2d871f48c554ea3cd886499d62de/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fb48801ec3533031d31ef03937cb4fba57325bb54a3db46140684fb12b06acfb/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1fecc2d9ed785302e25e086bf415154a6bf84e509831892351d1887821d878e5/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/95bb1a24c4c21522c0a6462ac8c45ec653eebc2f9e5ab317cca861f2062fb090/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e8cc6f64be916bf2db79bad71497aad0cac5f3f035de5dec377029f9c57bd536/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a6a755fb607bde47c4519060cd92476c5f707c65915d00a56106a3b5abee9210/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/d436be09595e8f66f90307e294c25a01383a9f2d8f0764a9e75c9e048a356111/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/164ebfb8c48d49bc37aa0fa6e62c0e16f4b586e6ddc4710073d8068dff5a3d2b/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/6930965ee61ced9a22dac02876236194be4669fe17792e0c582ed66c9dc93537/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/77e336d8d115ae9cc8ac107fbcad780560793cfa196924c513289dc7c8e72bbe/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/1e200c19cfe0df9e2e931ac2babf33ed5652d6c0e738cbf9667f88cbc655f6aa/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/fe46e85b5f1fd575d1860e95ae829c767bd4287c933ce93fb900bdc2d0901f97/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/6d52ff3a0ccdc4f021b41e69ef82181ef2b8df9c444cfbba60b83bd9ceef3b34/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/be25ac49550118567a0aedb5d531b0a8cc8215f0137cf3c7077e04f9b1a1ad9c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/a4f2dc737509e9e8558062d12141437f4a68054fcf69a18221bf4771f08c28cd/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/51bd790c0bc4ef6b31c08c3035c6970173908ed1e3be3ea121334c1947f86fce/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/49bac1f26192234c88a93085c6b4a9e09ca0db317752e1267faa6ed1d9de8cdf/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/da8ef9021bab05c7765bc98f979d18bd886cf5fc0f08bcd4aab1e706f2e4560e/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/861da098711247fabe4060cf92c56c9869f42d6071c7c78c8c37072d1cc8c94f/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/a5a4616ebac34935ca66322ea9275c3a763e12a04763ade4e47c45dec4b4499d/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/667339a001d624e2f45b7564c124ad3afad6cf3861347acfe32716932bbd2e51/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/6c693a0533ac9e7cf58891aec257bd77361642174c83ba90825f0fc06d6e9619/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/3cc0d3f7944626111920495e80a9345d0d119ada44090512b4e8dbfa56c84a2d/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/7390c161ccb23f4b9979c3d5b1d6c4ba4d0b7e445ec45686d670f1ad4c02d3e6/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/48b5921c5a61cc521b82740db787eda7469c968db8c481c47638e6a50038de0b/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/063129a6106466604029e8198a6c2b8736f293163c9fbd4d640ea31f42b6d53d/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/f82fc76d6e1dba4dbf6b823c3530f20b034f533af2826ece881a0d2b21f33129/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/a037b076ef1e414be343250099c5e41cddf54d3fd8261d7bb348ec203b8455f9/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/45471a1517cce9d9fc72b0bbaf59fdc94c6e5caaee00eb515bebe878b648c762/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8192eb2eb57717a873b655557ed72fc87bbd1c6e7cb37cf7d6f330b397e268f7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f6cd759ca106d3adb03db3c869fd6ab0051ccfebd5a2263ea3e37d02e6dfc432/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1b5c50a8ccb44f3a7a33269d2a379eda2d8f8cf546a8197a1f6f462a4f9f6b2a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/19a3071431d306e173ddc8c62277888e733366893bba8834bcf555bc01daad2a/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/878fe7648ac81aff34e1fdcde1fab324695ce3d3d7fc7e09e0cc5ad0c1fc3fa4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6cf65682c028050a260d9ca4c4c8373799f3837b00d95b788cfb51ff0e844de9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/115d0692196d98601d92bf6ce8267e63af4523326234db4d9e1e1014bf262bec/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/44c39c9dd8bbe8cea8bea8a1729f4405352017ec6e47a5ce5f62021d47a8b04a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f7e83c34835d306bda979bfb6cc11f62fed0ade234339ebde82e88d7dc297a9b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/46dfc8b537c67ae9dd2a221a9385248c247a0eff5ffa9099f83ca193423081dc/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863109 5039 manager.go:217] Machine: {Timestamp:2026-10-30 21:09:20.862668743 +0000 UTC m=+0.072608364 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201507 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c1040ef495967deb6b62a4a9811f SystemUUID:ec24c104-0ef4-9596-7deb-6b62a4a9811f BootID:d17a07ac-3230-469a-9b8f-d115bb44b42c Filesystems:[{Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a877cc0-db2f-42fc-9057-a62433401379/volumes/kubernetes.io~projected/kube-api-access-7x7bn DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3d488ee4d1084136914e8ed919fda8c8ceac2ca8f9d0dbebcde28d55f6da50e/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73dabeac41b907749eb08bad743aab89552a52fb54565a615e8003e955e6da50/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b8386a1-1b52-440d-b134-9c42390f70ba/volumes/kubernetes.io~projected/kube-api-access-5jfvg DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9feefbd2-6be8-47ab-a427-e9e91ff48e1a/volumes/kubernetes.io~projected/kube-api-access-6drzd DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9781bb5-7481-4803-8ef9-2eca6954c446/volumes/kubernetes.io~projected/kube-api-access-bkscn DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5abe8e87-10f7-453b-872f-ca366598dd92/volumes/kubernetes.io~projected/kube-api-access-7qwjr DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eae9a82fe50785ea85fb1db3de435afe0ae3e0fa2f94fbe0418a1485361b6fb9/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe/volumes/kubernetes.io~projected/kube-api-access-c4dgl DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75/volumes/kubernetes.io~projected/kube-api-access-zwc6h DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee11341a-54bb-412f-ad81-2c8ef689247a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/volumes/kubernetes.io~projected/kube-api-access-gm8rm DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/718af1425bb4fa3fc3cec8d0a38db129df76a08ad30cb5a49a0ac71f3c6c7aab/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a877cc0-db2f-42fc-9057-a62433401379/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56767bb662f4e6e3dc03bc2056bd224214d8e87117aa7d6674ec6351b8f16b82/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5/volumes/kubernetes.io~projected/kube-api-access-v2dj8 DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad49a05a52ced1cfe94cbebbd43a952e5d43555473c73f50bd8939d1e5193894/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a8e67f5-3ce1-412e-afb4-efbae8b2db98/volumes/kubernetes.io~projected/kube-api-access-6dn2f DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d900c012506d9ce2649bfe5419204d063aafe5f25b90b291508da8e18db8a74/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34/volumes/kubernetes.io~projected/kube-api-access-wmf2d DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/241c8c8035a644c1d27ea3b08471378c6e7a34eca10748f69565a49feadabb37/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffdf32f3-9268-4552-aa33-1b201380e88e/volumes/kubernetes.io~projected/kube-api-access-g7zwx DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c50ba7d148d07dd9a3ddad49d64f2c273bbeab7d05a43af91c890a3b0a1945e1/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8704650fadc22a639b786e09ec08fcb97dea25c18d5ea8f08e132d2ea7fe16c2/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~projected/kube-api-access-l2f2m DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08beba50-863a-4f30-a1ad-324cfde2159f/volumes/kubernetes.io~projected/kube-api-access-bwjnp DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~projected/kube-api-access-bsgr4 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volumes/kubernetes.io~projected/kube-api-access-tqbks DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16ec93c8d403242673993dc086e383b5bb76d9b8f69f60c7f2d5f99b6f40fb1e/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/95dba424-9ecd-43b9-a09e-8c643019e969/volumes/kubernetes.io~projected/kube-api-access-gcnsn DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1e1f33-a5b9-442c-8589-970667a86890/volumes/kubernetes.io~projected/kube-api-access-ncd4q DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad5e0914-9ca0-4dbf-952e-1b9812908b1c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2459def9-9723-4de1-9a93-5a2dfb7a3cca/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b3525026d116424ba68c89e44306b992182460015bf1de7199480c9bde41f6c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes/kubernetes.io~projected/kube-api-access-jwk4q DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4039f00544feb1eecd65f3c8ea6dbd23196bb56d4eb107e99c4ffdc7aa87d2ad/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e/volumes/kubernetes.io~projected/kube-api-access-bmlpj DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d18517ffd425309b618710c74180cd5e1ec7919b9c61ac4043ad36f9ff3bb4b5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9abec43900272a6f0bdb2f7dc1d02eac6471b47c58815b041d5f6812c11680d9/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c24f04d8515b1cff4df3aa8f22a5dcd5abcdfc509065cc61fb3a35bd39c76d61/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4142b823-6b14-49f2-bf14-e8231f9296f3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b9b6d2f2a0e06cae868dfdbdef26ab65a7a97cd215208f88454124c6d8006bf/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eea00f30ee1404023e5763d857c4415acdf543ee6321c7f036c81bdd3588cfa6/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/106e7cc6-df7c-4e5a-8876-1c0456a35023/volumes/kubernetes.io~projected/kube-api-access-jw94c DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08beba50-863a-4f30-a1ad-324cfde2159f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~projected/kube-api-access-q4zh2 DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21966c11-af60-40d2-bde4-d6cb8296cad4/volumes/kubernetes.io~projected/kube-api-access-tgbcm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab2c2178540afbea79f7bf01d40d098687301535a6d26ee1eaf8227203bcfc00/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4/volumes/kubernetes.io~projected/kube-api-access-bvkt5 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b13507f7-c09f-46bd-b76b-50ba62300dfb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5b111d-8074-4bda-8ac9-7d1f1f623c59/volumes/kubernetes.io~projected/kube-api-access-4wmlw DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9771e8b88f6bc119695aa5539bf40093ff30c2dfa82b73e9024359074099a1a/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84eda726-2f51-4c10-a829-681fdbb559e2/volumes/kubernetes.io~projected/kube-api-access-56rwq DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c/volumes/kubernetes.io~projected/kube-api-access-l656s DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c7fd700-057c-4ceb-88c4-01620fa6329d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee11341a-54bb-412f-ad81-2c8ef689247a/volumes/kubernetes.io~projected/kube-api-access-2xhlv DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d887faa8-91ab-4a8e-8b92-38a199ef7081/volumes/kubernetes.io~projected/kube-api-access-6zhcn DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/209912e2-5544-4a98-9282-ed7e28c449d0/volumes/kubernetes.io~projected/kube-api-access-v4cnl DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8887f474ad0e44a6c4a02a4468726a925239b9fc06f3b4657c4e46b738ae6a28/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9931cd13-1775-4f3d-9300-d2cb16df7acf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84eda726-2f51-4c10-a829-681fdbb559e2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d98196c76b3b2cb7186584bd305ce10a7e852d1b732de529162db7d320de6691/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7d9141568c42fde15f5e64a81efffa2b58b10ce4ef5d4a0213878a91069b1c5/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9931cd13-1775-4f3d-9300-d2cb16df7acf/volumes/kubernetes.io~projected/kube-api-access-mwg9v DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16ec93c8d403242 MacAddress:8a:0a:7b:9d:30:89 Speed:10000 Mtu:8901} {Name:241c8c8035a644c MacAddress:be:79:1f:ee:5d:d5 Speed:10000 Mtu:8901} {Name:4039f00544feb1e MacAddress:86:26:c4:88:9c:0a Speed:10000 Mtu:8901} {Name:4b9b6d2f2a0e06c MacAddress:fa:4f:96:04:61:48 Speed:10000 Mtu:8901} {Name:56767bb662f4e6e MacAddress:32:40:66:b9:c7:fc Speed:10000 Mtu:8901} {Name:9abec43900272a6 MacAddress:52:40:0f:6e:09:08 Speed:10000 Mtu:8901} {Name:a3d488ee4d10841 MacAddress:8a:7e:7b:4d:11:66 Speed:10000 Mtu:8901} {Name:ab2c2178540afbe MacAddress:46:33:0c:5c:7f:73 Speed:10000 Mtu:8901} {Name:ad49a05a52ced1c MacAddress:96:a5:7e:2f:dc:ca Speed:10000 Mtu:8901} {Name:b05f5e8f5b318a1 MacAddress:4e:83:3a:d0:57:e4 Speed:10000 Mtu:8901} {Name:b9771e8b88f6bc1 MacAddress:1a:42:a6:ad:16:5a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e3:93:51:1d Speed:0 Mtu:9001} {Name:br-int MacAddress:96:96:fc:e1:21:8f Speed:0 Mtu:8901} {Name:c24f04d8515b1cf MacAddress:3e:05:11:60:83:07 Speed:10000 Mtu:8901} {Name:c50ba7d148d07dd MacAddress:42:c9:49:7a:75:8c Speed:10000 Mtu:8901} {Name:d18517ffd425309 MacAddress:ba:5c:30:7b:a1:83 Speed:10000 Mtu:8901} {Name:eae9a82fe50785e MacAddress:c6:16:16:69:5e:7a Speed:10000 Mtu:8901} {Name:eea00f30ee14040 MacAddress:86:cb:f5:3b:ff:e0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e3:93:51:1d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:f9:3c:6c:19:0c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:10:c5:3e:bd:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863467 5039 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863512 5039 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863777 5039 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863797 5039 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863915 5039 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863922 5039 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863932 5039 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863946 5039 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.863965 5039 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864100 5039 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864194 5039 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864224 5039 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864243 5039 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864256 5039 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864265 5039 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.864273 5039 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.865207 5039 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.865221 5039 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:09:20.866023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.865705 5039 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:09:20.868452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868433 5039 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:09:20.868452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868448 5039 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:09:20.868605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868591 5039 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:09:20.868815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868800 5039 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:09:20.868896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868840 5039 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:09:20.868966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868955 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:09:20.869018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868975 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:09:20.869018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868986 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:09:20.869018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.868996 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:09:20.869018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869005 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869026 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869036 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869045 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869057 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869066 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869086 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869101 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869136 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:09:20.869186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.869147 5039 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:09:20.872659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.872642 5039 server.go:1263] "Started kubelet" Oct 30 21:09:20.872743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.872696 5039 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:09:20.873084 ip-10-0-133-149 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:09:20.873308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.872737 5039 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:09:20.873493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.873317 5039 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:09:20.873698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.873671 5039 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:09:20.876818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.875986 5039 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:09:20.878095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878076 5039 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:09:20.878201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878186 5039 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:09:20.878267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878244 5039 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:03:43 +0000 UTC" deadline="2028-07-23 15:30:20.56189267 +0000 UTC" Oct 30 21:09:20.878267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878262 5039 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15162h20m59.683633242s" Oct 30 21:09:20.878365 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:20.878294 5039 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-149.ec2.internal\" not found" Oct 30 21:09:20.878365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878325 5039 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:09:20.878365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878332 5039 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:09:20.878558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.878543 5039 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:09:20.880672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880643 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9781bb5-7481-4803-8ef9-2eca6954c446" volumeName="kubernetes.io/projected/d9781bb5-7481-4803-8ef9-2eca6954c446-kube-api-access-bkscn" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880674 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880692 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e" volumeName="kubernetes.io/projected/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-kube-api-access-bmlpj" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880704 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2459def9-9723-4de1-9a93-5a2dfb7a3cca" volumeName="kubernetes.io/secret/2459def9-9723-4de1-9a93-5a2dfb7a3cca-agent-certs" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880718 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880730 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c7fd700-057c-4ceb-88c4-01620fa6329d" volumeName="kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-default-certificate" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880741 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9931cd13-1775-4f3d-9300-d2cb16df7acf" volumeName="kubernetes.io/projected/9931cd13-1775-4f3d-9300-d2cb16df7acf-kube-api-access-mwg9v" seLinuxMountContext="" Oct 30 21:09:20.880767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880754 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bafe23a-4bb7-477d-81a1-6024f5c3d9d4" volumeName="kubernetes.io/empty-dir/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-tmp-dir" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880769 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880782 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce09dc0-573a-4ac2-a6aa-6a37bead48fe" volumeName="kubernetes.io/projected/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe-kube-api-access-c4dgl" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880795 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b13507f7-c09f-46bd-b76b-50ba62300dfb" volumeName="kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-env-overrides" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880807 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b13507f7-c09f-46bd-b76b-50ba62300dfb" volumeName="kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-config" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880819 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880831 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c7fd700-057c-4ceb-88c4-01620fa6329d" volumeName="kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-stats-auth" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880846 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dba424-9ecd-43b9-a09e-8c643019e969" volumeName="kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880865 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dba424-9ecd-43b9-a09e-8c643019e969" volumeName="kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880878 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad5e0914-9ca0-4dbf-952e-1b9812908b1c" volumeName="kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-tuned" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880893 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08beba50-863a-4f30-a1ad-324cfde2159f" volumeName="kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-snapshots" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880908 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5b111d-8074-4bda-8ac9-7d1f1f623c59" volumeName="kubernetes.io/empty-dir/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-tmp-dir" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880921 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5" volumeName="kubernetes.io/configmap/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-config" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880936 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880947 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee11341a-54bb-412f-ad81-2c8ef689247a" volumeName="kubernetes.io/secret/ee11341a-54bb-412f-ad81-2c8ef689247a-cert" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880961 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f71a2be6-1f45-43a6-ae97-8ebd007341fd" volumeName="kubernetes.io/configmap/f71a2be6-1f45-43a6-ae97-8ebd007341fd-nginx-conf" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.880973 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad5e0914-9ca0-4dbf-952e-1b9812908b1c" volumeName="kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-tmp" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881009 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c837b6b5-8ffc-4265-8d5c-1cc96e97fe75" volumeName="kubernetes.io/projected/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75-kube-api-access-zwc6h" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881027 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d887faa8-91ab-4a8e-8b92-38a199ef7081" volumeName="kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881044 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5b111d-8074-4bda-8ac9-7d1f1f623c59" volumeName="kubernetes.io/configmap/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-config-volume" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881057 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a877cc0-db2f-42fc-9057-a62433401379" volumeName="kubernetes.io/secret/8a877cc0-db2f-42fc-9057-a62433401379-serving-cert" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881075 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dba424-9ecd-43b9-a09e-8c643019e969" volumeName="kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-binary-copy" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881088 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b13507f7-c09f-46bd-b76b-50ba62300dfb" volumeName="kubernetes.io/secret/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:09:20.881095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881101 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21966c11-af60-40d2-bde4-d6cb8296cad4" volumeName="kubernetes.io/projected/21966c11-af60-40d2-bde4-d6cb8296cad4-kube-api-access-tgbcm" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881112 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d887faa8-91ab-4a8e-8b92-38a199ef7081" volumeName="kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881124 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5" volumeName="kubernetes.io/projected/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-kube-api-access-v2dj8" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881136 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881149 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881175 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c7fd700-057c-4ceb-88c4-01620fa6329d" volumeName="kubernetes.io/projected/7c7fd700-057c-4ceb-88c4-01620fa6329d-kube-api-access-q4zh2" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881187 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2459def9-9723-4de1-9a93-5a2dfb7a3cca" volumeName="kubernetes.io/configmap/2459def9-9723-4de1-9a93-5a2dfb7a3cca-konnectivity-ca" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881196 5039 factory.go:55] Registering systemd factory Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881213 5039 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881198 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4142b823-6b14-49f2-bf14-e8231f9296f3" volumeName="kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881269 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abe8e87-10f7-453b-872f-ca366598dd92" volumeName="kubernetes.io/projected/5abe8e87-10f7-453b-872f-ca366598dd92-kube-api-access-7qwjr" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881290 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9feefbd2-6be8-47ab-a427-e9e91ff48e1a" volumeName="kubernetes.io/projected/9feefbd2-6be8-47ab-a427-e9e91ff48e1a-kube-api-access-6drzd" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881303 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b13507f7-c09f-46bd-b76b-50ba62300dfb" volumeName="kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881319 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d887faa8-91ab-4a8e-8b92-38a199ef7081" volumeName="kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881330 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881342 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08beba50-863a-4f30-a1ad-324cfde2159f" volumeName="kubernetes.io/secret/08beba50-863a-4f30-a1ad-324cfde2159f-serving-cert" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881355 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881366 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84eda726-2f51-4c10-a829-681fdbb559e2" volumeName="kubernetes.io/configmap/84eda726-2f51-4c10-a829-681fdbb559e2-signing-cabundle" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881375 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9931cd13-1775-4f3d-9300-d2cb16df7acf" volumeName="kubernetes.io/configmap/9931cd13-1775-4f3d-9300-d2cb16df7acf-config" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881383 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="209912e2-5544-4a98-9282-ed7e28c449d0" volumeName="kubernetes.io/projected/209912e2-5544-4a98-9282-ed7e28c449d0-kube-api-access-v4cnl" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881412 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881422 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a8e67f5-3ce1-412e-afb4-efbae8b2db98" volumeName="kubernetes.io/projected/3a8e67f5-3ce1-412e-afb4-efbae8b2db98-kube-api-access-6dn2f" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881434 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5b111d-8074-4bda-8ac9-7d1f1f623c59" volumeName="kubernetes.io/projected/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-kube-api-access-4wmlw" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881452 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" volumeName="kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-trusted-ca" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881460 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9931cd13-1775-4f3d-9300-d2cb16df7acf" volumeName="kubernetes.io/secret/9931cd13-1775-4f3d-9300-d2cb16df7acf-serving-cert" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881468 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad5e0914-9ca0-4dbf-952e-1b9812908b1c" volumeName="kubernetes.io/projected/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-kube-api-access-l2f2m" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881475 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c" volumeName="kubernetes.io/projected/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-kube-api-access-l656s" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881483 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08beba50-863a-4f30-a1ad-324cfde2159f" volumeName="kubernetes.io/projected/08beba50-863a-4f30-a1ad-324cfde2159f-kube-api-access-bwjnp" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881499 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="106e7cc6-df7c-4e5a-8876-1c0456a35023" volumeName="kubernetes.io/projected/106e7cc6-df7c-4e5a-8876-1c0456a35023-kube-api-access-jw94c" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881506 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21966c11-af60-40d2-bde4-d6cb8296cad4" volumeName="kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-daemon-config" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881513 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84eda726-2f51-4c10-a829-681fdbb559e2" volumeName="kubernetes.io/projected/84eda726-2f51-4c10-a829-681fdbb559e2-kube-api-access-56rwq" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881519 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84eda726-2f51-4c10-a829-681fdbb559e2" volumeName="kubernetes.io/secret/84eda726-2f51-4c10-a829-681fdbb559e2-signing-key" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881526 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" volumeName="kubernetes.io/projected/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-kube-api-access-gm8rm" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881533 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d887faa8-91ab-4a8e-8b92-38a199ef7081" volumeName="kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881540 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08beba50-863a-4f30-a1ad-324cfde2159f" volumeName="kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881548 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b8386a1-1b52-440d-b134-9c42390f70ba" volumeName="kubernetes.io/projected/2b8386a1-1b52-440d-b134-9c42390f70ba-kube-api-access-5jfvg" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881555 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f25c0ef-fd48-4bf6-b5a9-0261fe507a34" volumeName="kubernetes.io/projected/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34-kube-api-access-wmf2d" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881569 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a877cc0-db2f-42fc-9057-a62433401379" volumeName="kubernetes.io/configmap/8a877cc0-db2f-42fc-9057-a62433401379-config" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881575 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b13507f7-c09f-46bd-b76b-50ba62300dfb" volumeName="kubernetes.io/projected/b13507f7-c09f-46bd-b76b-50ba62300dfb-kube-api-access-tqbks" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881583 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccf83b0-6149-4f2b-b3b1-db339a5a245e" volumeName="kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881590 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdf32f3-9268-4552-aa33-1b201380e88e" volumeName="kubernetes.io/empty-dir/ffdf32f3-9268-4552-aa33-1b201380e88e-data-volume" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881598 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e" volumeName="kubernetes.io/configmap/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-serviceca" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881605 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08beba50-863a-4f30-a1ad-324cfde2159f" volumeName="kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-tmp" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881616 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881626 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a877cc0-db2f-42fc-9057-a62433401379" volumeName="kubernetes.io/projected/8a877cc0-db2f-42fc-9057-a62433401379-kube-api-access-7x7bn" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881636 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" volumeName="kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-config" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881644 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bafe23a-4bb7-477d-81a1-6024f5c3d9d4" volumeName="kubernetes.io/projected/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-kube-api-access-bvkt5" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881651 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdf32f3-9268-4552-aa33-1b201380e88e" volumeName="kubernetes.io/projected/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-api-access-g7zwx" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881658 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1e1f33-a5b9-442c-8589-970667a86890" volumeName="kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881665 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21966c11-af60-40d2-bde4-d6cb8296cad4" volumeName="kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-cni-binary-copy" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881672 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dba424-9ecd-43b9-a09e-8c643019e969" volumeName="kubernetes.io/projected/95dba424-9ecd-43b9-a09e-8c643019e969-kube-api-access-gcnsn" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881679 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee11341a-54bb-412f-ad81-2c8ef689247a" volumeName="kubernetes.io/projected/ee11341a-54bb-412f-ad81-2c8ef689247a-kube-api-access-2xhlv" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881687 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" volumeName="kubernetes.io/secret/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-serving-cert" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881694 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881701 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57293ead-fcc2-47e7-8033-d0beec008c51" volumeName="kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881709 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08beba50-863a-4f30-a1ad-324cfde2159f" volumeName="kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-service-ca-bundle" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881716 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c" volumeName="kubernetes.io/configmap/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881724 5039 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abe8e87-10f7-453b-872f-ca366598dd92" volumeName="kubernetes.io/configmap/5abe8e87-10f7-453b-872f-ca366598dd92-telemetry-config" seLinuxMountContext="" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881732 5039 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:09:20.881983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.881737 5039 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:09:20.883967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.883325 5039 factory.go:153] Registering CRI-O factory Oct 30 21:09:20.883967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.883338 5039 factory.go:223] Registration of the crio container factory successfully Oct 30 21:09:20.883967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.883385 5039 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:09:20.883967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.883420 5039 factory.go:103] Registering Raw factory Oct 30 21:09:20.883967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.883434 5039 manager.go:1199] Started watching for new ooms in manager Oct 30 21:09:20.883967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.883811 5039 manager.go:319] Starting recovery of all containers Oct 30 21:09:20.899102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.899060 5039 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:09:20.899975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.899951 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:09:20.900596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.900494 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:09:20.913344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.913320 5039 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:09:20.913344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.913345 5039 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:09:20.913517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.913367 5039 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:09:20.913517 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:20.913419 5039 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:09:20.923151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.923125 5039 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:09:20.923295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.923273 5039 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:09:20.923966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.923948 5039 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:09:20.934484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.934468 5039 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:09:20.941280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.941259 5039 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:09:20.953245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.953218 5039 generic.go:356] "Generic (PLEG): container finished" podID="9ad7dad2adb3692daf79090c2b068e15" containerID="501e50849ed3267c04a07d21b78ee1600a8319f2ed8a143c7327f2afbe420261" exitCode=0 Oct 30 21:09:20.963320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.963297 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:09:20.963434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.963329 5039 generic.go:356] "Generic (PLEG): container finished" podID="8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf" containerID="5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb" exitCode=255 Oct 30 21:09:20.970519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.970496 5039 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="a2786a39b0fdb26003e15b6ca2e2e49fac71bf0a901d7226f670f2c079019a03" exitCode=0 Oct 30 21:09:20.970519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.970516 5039 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="00f98d9c6dc7a0b88c9d9d50e28417046e023370c1ac43960d6850fef7f0b32e" exitCode=0 Oct 30 21:09:20.970519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.970523 5039 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="890d117ec1b9c614be27533c3a4f2c4307ce00d193b1e1c76e8ee398d626c778" exitCode=0 Oct 30 21:09:20.970663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.970528 5039 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="486b18bee5f348d0a6ac9a360f105f9ba3c6b2f02542eba8f928880a89219080" exitCode=0 Oct 30 21:09:20.970663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.970534 5039 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="147d743c9f41c00f0ac1a0d285fb94cfe6464a7f8e340ff47b5e99f1d5ab928d" exitCode=0 Oct 30 21:09:20.970663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.970538 5039 generic.go:356] "Generic (PLEG): container finished" podID="95dba424-9ecd-43b9-a09e-8c643019e969" containerID="517b6558187902b51db47c16375ce934b222711465d832f33e82ac48002fc61b" exitCode=0 Oct 30 21:09:20.983786 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:20.983769 5039 manager.go:324] Recovery completed Oct 30 21:09:21.005217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005168 5039 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:09:21.005217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005182 5039 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:09:21.005217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005198 5039 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:09:21.005352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005305 5039 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:09:21.005352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005315 5039 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:09:21.005352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005331 5039 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:09:21.005352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005336 5039 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:09:21.005352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005341 5039 policy_none.go:50] "Start" Oct 30 21:09:21.005352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005349 5039 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:09:21.005552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005358 5039 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:09:21.005552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005489 5039 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:09:21.005552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005496 5039 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:09:21.005552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.005502 5039 policy_none.go:44] "Start" Oct 30 21:09:21.009708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.009685 5039 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:09:21.012204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.012192 5039 manager.go:341] "Starting Device Plugin manager" Oct 30 21:09:21.012772 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:21.012216 5039 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:09:21.012772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.012774 5039 server.go:87] "Starting device plugin registration server" Oct 30 21:09:21.013018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013006 5039 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:09:21.013065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013019 5039 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:09:21.013154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013125 5039 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:09:21.013274 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013239 5039 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:09:21.013331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013284 5039 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:09:21.013331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013292 5039 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:09:21.013441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013343 5039 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:09:21.013272703Z","UUID":"9ec53d77-251c-4810-a66f-49904c121510","Handler":null,"Name":"","Endpoint":""} Oct 30 21:09:21.013661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.013620 5039 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal"] Oct 30 21:09:21.014671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.014602 5039 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:09:21.014671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.014635 5039 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:09:21.014835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.014735 5039 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.015677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.015590 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" event={"ID":"524d7c023acdf7f01932ca99a8915366","Type":"ContainerStarted","Data":"096528ed2f68c12793386d1ca1d901949a308f5c6eae25709168f8cb79e887f8"} Oct 30 21:09:21.015823 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.015800 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" event={"ID":"524d7c023acdf7f01932ca99a8915366","Type":"ContainerStarted","Data":"6b4baac804c64adc3bf0a4e1515e3ed16235e923a2994879cda80ba4103fd9cd"} Oct 30 21:09:21.015907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.015829 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"9ad7dad2adb3692daf79090c2b068e15","Type":"ContainerStarted","Data":"08451d328bf52377b9bedef01e5778f53bc50649a1e17684218f7276391d90c7"} Oct 30 21:09:21.015907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.015869 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"9ad7dad2adb3692daf79090c2b068e15","Type":"ContainerDied","Data":"501e50849ed3267c04a07d21b78ee1600a8319f2ed8a143c7327f2afbe420261"} Oct 30 21:09:21.015907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.015888 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" event={"ID":"9ad7dad2adb3692daf79090c2b068e15","Type":"ContainerStarted","Data":"e18d24df8b15dbde531d24e0943d450a7e9cc9d9396f73368be24b9e8faff73e"} Oct 30 21:09:21.016154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.016136 5039 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.016849 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:21.016828 5039 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:09:21.036055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.036038 5039 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:09:21.036144 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:21.036078 5039 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.036723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.036708 5039 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:09:21.036765 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:21.036746 5039 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.113410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.113374 5039 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:09:21.114656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.114636 5039 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:09:21.114727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.114673 5039 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:09:21.114727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.114687 5039 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-149.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:09:21.114782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.114734 5039 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.123982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.123959 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.124036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.123988 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.124036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.124007 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/524d7c023acdf7f01932ca99a8915366-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"524d7c023acdf7f01932ca99a8915366\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.143925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.143906 5039 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.143994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.143980 5039 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.224668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.224636 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.224809 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.224762 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.224855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.224797 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/524d7c023acdf7f01932ca99a8915366-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"524d7c023acdf7f01932ca99a8915366\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.224855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.224727 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.224929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.224858 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad7dad2adb3692daf79090c2b068e15-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" (UID: \"9ad7dad2adb3692daf79090c2b068e15\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.224929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.224861 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/524d7c023acdf7f01932ca99a8915366-config\") pod \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" (UID: \"524d7c023acdf7f01932ca99a8915366\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.865418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.865379 5039 apiserver.go:52] "Watching apiserver" Oct 30 21:09:21.907809 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.907776 5039 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:09:21.908884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.908847 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-psc8l","kube-system/konnectivity-agent-hsqfc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4","openshift-network-operator/iptables-alerter-r6tmj","openshift-ingress/router-default-77b4b6cbf9-7xzk8","openshift-monitoring/cluster-monitoring-operator-884576c57-xr594","kube-system/kas-connection-checker-685895c7c9-bnjl2","openshift-multus/network-metrics-daemon-c88gd","kube-system/global-pull-secret-syncer-b9l4r","openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq","openshift-insights/periodic-gathering-vb2td-csvl7","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn","openshift-multus/multus-kzg7s","kube-system/kas-connection-checker-685895c7c9-kmzkx","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc","openshift-image-registry/image-registry-5b65c6946-57jbj","openshift-multus/multus-additional-cni-plugins-cf9xp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f","openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch","openshift-network-console/networking-console-plugin-86c478bb88-phllg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal","openshift-network-diagnostics/network-check-target-85c8w","openshift-console-operator/console-operator-7cd56b8dbc-gh745","openshift-insights/insights-runtime-extractor-sq4d5","openshift-image-registry/node-ca-87n87","openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg","openshift-cluster-node-tuning-operator/tuned-s586p","openshift-service-ca/service-ca-5fffc97bdf-hh45f","openshift-ingress-canary/ingress-canary-5lcrb","kube-system/kas-connection-checker-685895c7c9-j9ct9","openshift-insights/insights-operator-7c4986d449-vws6h","openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w","openshift-dns/dns-default-rp8hx","openshift-dns/node-resolver-c7kgv","openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:09:21.912174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.912152 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:21.912547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.912529 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:21.913020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.913001 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:21.917073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.917048 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:21.917461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.917432 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:21.919900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.919881 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:21.921357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.921337 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.921462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.921337 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.922062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.922040 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.922272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.922255 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.922873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.922846 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:09:21.922969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.922894 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-z65n7\"" Oct 30 21:09:21.922969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.922918 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5rj44\"" Oct 30 21:09:21.923072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.922997 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xsb5g\"" Oct 30 21:09:21.923230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923188 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.923466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923445 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:09:21.923547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923532 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.923668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923544 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.923668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923551 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.923668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923450 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:09:21.923668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923594 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:09:21.923859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923769 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.923859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923818 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:09:21.923953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923890 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:09:21.923953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.923938 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:09:21.924075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924055 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42gzh\"" Oct 30 21:09:21.924172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924085 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.924172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924127 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:09:21.924172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924157 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:09:21.924372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924195 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gtfcx\"" Oct 30 21:09:21.924372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924276 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:09:21.924669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.924655 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:21.928094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.928077 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:21.928173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.928120 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:09:21.928339 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.928324 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7hhvb\"" Oct 30 21:09:21.928451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.928439 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mcwp8\"" Oct 30 21:09:21.928524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.928469 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fs65v\"" Oct 30 21:09:21.928606 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.928595 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.929136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.929122 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:09:21.929207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.929151 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.929774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.929756 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.929945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.929930 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:09:21.930057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.930041 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.930544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.930530 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:21.932313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.932294 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.932417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.932301 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d8vcg\"" Oct 30 21:09:21.932417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.932303 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.933027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933012 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.933110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933028 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:09:21.933110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933030 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qbpqm\"" Oct 30 21:09:21.933246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933231 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:09:21.933299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933255 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:09:21.933299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933268 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.933299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933290 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.933299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933274 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:09:21.933451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.933286 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.938055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.938038 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.939368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.939349 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g2wf2\"" Oct 30 21:09:21.939556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.939538 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:09:21.942956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.942931 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:09:21.943054 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.942994 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:09:21.943054 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943037 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:09:21.943327 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943308 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.943422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943343 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.943536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943515 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:09:21.943591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943576 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\"" Oct 30 21:09:21.943639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943611 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.943702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.943656 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hnk74\"" Oct 30 21:09:21.944084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944063 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:09:21.944188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944171 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.944349 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944326 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:09:21.944502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944476 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:09:21.944604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944561 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:09:21.944727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944621 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:09:21.944774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944760 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:09:21.944825 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944773 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:09:21.944904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.944884 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.945554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945513 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:09:21.945845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945582 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.945845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945837 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5899\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945840 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945839 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945846 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945832 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8m2s\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945884 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945910 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.945969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945889 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.946255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.945984 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:09:21.946255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946005 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2tg5\"" Oct 30 21:09:21.946255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946045 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:09:21.946255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946164 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d6fgf\"" Oct 30 21:09:21.946455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946255 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jv27h\"" Oct 30 21:09:21.946455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946344 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.946596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946529 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:09:21.946596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946574 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\"" Oct 30 21:09:21.946686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946623 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\"" Oct 30 21:09:21.946686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946664 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:09:21.946779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946716 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:09:21.946779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946765 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.946888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946870 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t284x\"" Oct 30 21:09:21.946888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946877 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nxx4w\"" Oct 30 21:09:21.947003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946925 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.947003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946937 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.947003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.946964 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjvtw\"" Oct 30 21:09:21.947290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.947275 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:09:21.947718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.947706 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.947801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.947786 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:09:21.947897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.947862 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w6j9l\"" Oct 30 21:09:21.947954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.947909 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:09:21.948013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.947995 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.948062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948016 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6qvgx\"" Oct 30 21:09:21.948110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948069 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:09:21.948157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948127 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.948205 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948127 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:09:21.948266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948251 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kmcds\"" Oct 30 21:09:21.948310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948265 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tkfgn\"" Oct 30 21:09:21.948310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948276 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:09:21.948310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948302 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:09:21.948477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948347 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.948477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948401 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:09:21.948571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948535 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:09:21.948613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948577 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:09:21.948822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.948806 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8rdf8\"" Oct 30 21:09:21.974107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.974086 5039 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.974256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.974240 5039 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.981496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.981475 5039 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:09:21.993793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.993749 5039 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:09:21.993793 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:21.993788 5039 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-149.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-149.ec2.internal" Oct 30 21:09:21.994155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:21.994141 5039 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:09:21.994189 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:21.994179 5039 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-149.ec2.internal" Oct 30 21:09:22.026569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.026541 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.029502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029481 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.029559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029513 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.029559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029531 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.029559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029548 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.029701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029569 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f71a2be6-1f45-43a6-ae97-8ebd007341fd-nginx-conf\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:22.029701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029643 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:22.029701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029678 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-system-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.029826 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029730 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.029826 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029809 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xhlv\" (UniqueName: \"kubernetes.io/projected/ee11341a-54bb-412f-ad81-2c8ef689247a-kube-api-access-2xhlv\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:22.029972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029852 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zh2\" (UniqueName: \"kubernetes.io/projected/7c7fd700-057c-4ceb-88c4-01620fa6329d-kube-api-access-q4zh2\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.029972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029889 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnsn\" (UniqueName: \"kubernetes.io/projected/95dba424-9ecd-43b9-a09e-8c643019e969-kube-api-access-gcnsn\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.029972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029918 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f71a2be6-1f45-43a6-ae97-8ebd007341fd-nginx-conf\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:22.029972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029921 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.029972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029952 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.029972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029968 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/84eda726-2f51-4c10-a829-681fdbb559e2-signing-key\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:22.030184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.029999 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-etc-kubernetes\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.030184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030027 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.030184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030147 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbcm\" (UniqueName: \"kubernetes.io/projected/21966c11-af60-40d2-bde4-d6cb8296cad4-kube-api-access-tgbcm\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.030184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030179 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-env-overrides\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.030334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030210 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.030334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030253 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drzd\" (UniqueName: \"kubernetes.io/projected/9feefbd2-6be8-47ab-a427-e9e91ff48e1a-kube-api-access-6drzd\") pod \"migrator-66df5d496b-lxmgq\" (UID: \"9feefbd2-6be8-47ab-a427-e9e91ff48e1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:22.030439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030364 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/84eda726-2f51-4c10-a829-681fdbb559e2-signing-key\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:22.030439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030380 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-modprobe-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.030515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030439 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f2m\" (UniqueName: \"kubernetes.io/projected/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-kube-api-access-l2f2m\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.030515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030502 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-config-volume\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.030578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030542 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-env-overrides\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.030578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030561 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-k8s-cni-cncf-io\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.030654 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030610 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.030702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030668 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-serviceca\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.030755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030715 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc6h\" (UniqueName: \"kubernetes.io/projected/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75-kube-api-access-zwc6h\") pod \"kas-connection-checker-685895c7c9-j9ct9\" (UID: \"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75\") " pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:22.030803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030753 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.030803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030754 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-config-volume\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.030803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030788 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw94c\" (UniqueName: \"kubernetes.io/projected/106e7cc6-df7c-4e5a-8876-1c0456a35023-kube-api-access-jw94c\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:22.030942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030815 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkscn\" (UniqueName: \"kubernetes.io/projected/d9781bb5-7481-4803-8ef9-2eca6954c446-kube-api-access-bkscn\") pod \"kas-connection-checker-685895c7c9-bnjl2\" (UID: \"d9781bb5-7481-4803-8ef9-2eca6954c446\") " pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:22.030942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030913 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-config\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.031030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030954 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a877cc0-db2f-42fc-9057-a62433401379-serving-cert\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:22.031030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.030971 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-lib-modules\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.031114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031031 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34-kube-api-access-wmf2d\") pod \"network-check-source-6f78d59f99-dq7ch\" (UID: \"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:22.031114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031035 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-serviceca\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.031172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031098 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-cnibin\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.031172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031130 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a877cc0-db2f-42fc-9057-a62433401379-serving-cert\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:22.031227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031212 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l656s\" (UniqueName: \"kubernetes.io/projected/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-kube-api-access-l656s\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.031261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031242 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-config\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.031261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031246 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-bin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.031316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031279 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-trusted-ca\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.031344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031304 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-kubelet\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.031372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031352 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-systemd-units\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.031473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031372 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.031473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031413 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-script-lib\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.031628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031593 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.031676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031602 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-script-lib\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.031676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031624 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-trusted-ca\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.031760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031678 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmlw\" (UniqueName: \"kubernetes.io/projected/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-kube-api-access-4wmlw\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.031760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031703 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.031760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031733 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.031866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031766 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.031866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031821 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.031866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031852 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwg9v\" (UniqueName: \"kubernetes.io/projected/9931cd13-1775-4f3d-9300-d2cb16df7acf-kube-api-access-mwg9v\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:22.031999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031880 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-host\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.031999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031885 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.031999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031921 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.032134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.031980 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-multus-certs\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.032184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032153 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-sys\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.032234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032223 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08beba50-863a-4f30-a1ad-324cfde2159f-serving-cert\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.032282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032254 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwjr\" (UniqueName: \"kubernetes.io/projected/5abe8e87-10f7-453b-872f-ca366598dd92-kube-api-access-7qwjr\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.032336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032281 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.032336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032303 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-netd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.032336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032320 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.032476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032342 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/84eda726-2f51-4c10-a829-681fdbb559e2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:22.032476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032369 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-tmp\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.032476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032409 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee11341a-54bb-412f-ad81-2c8ef689247a-cert\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:22.032476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032433 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jfvg\" (UniqueName: \"kubernetes.io/projected/2b8386a1-1b52-440d-b134-9c42390f70ba-kube-api-access-5jfvg\") pod \"volume-data-source-validator-58c9f5f46d-48pxc\" (UID: \"2b8386a1-1b52-440d-b134-9c42390f70ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:22.032667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032480 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8rm\" (UniqueName: \"kubernetes.io/projected/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-kube-api-access-gm8rm\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.032667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032589 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-binary-copy\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.032667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032661 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-daemon-config\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.032811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032707 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-kubernetes\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.032811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032722 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-tmp\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.032811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032773 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee11341a-54bb-412f-ad81-2c8ef689247a-cert\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:22.032946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032816 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08beba50-863a-4f30-a1ad-324cfde2159f-serving-cert\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.032946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032901 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.033045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032939 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cnl\" (UniqueName: \"kubernetes.io/projected/209912e2-5544-4a98-9282-ed7e28c449d0-kube-api-access-v4cnl\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.033045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.032996 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/84eda726-2f51-4c10-a829-681fdbb559e2-signing-cabundle\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:22.033045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033006 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.033178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033060 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-default-certificate\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.033178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033084 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-daemon-config\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.033286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033191 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-etc-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.033286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033221 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-node-log\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.033286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033268 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.033286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033275 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-default-certificate\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.033286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033273 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdf32f3-9268-4552-aa33-1b201380e88e-data-volume\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033303 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a877cc0-db2f-42fc-9057-a62433401379-config\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033322 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdf32f3-9268-4552-aa33-1b201380e88e-data-volume\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033322 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-host-slash\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033351 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-kubelet\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033437 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-hosts-file\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033478 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.033538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033502 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a877cc0-db2f-42fc-9057-a62433401379-config\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:22.033835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033597 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9931cd13-1775-4f3d-9300-d2cb16df7acf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:22.033835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033632 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-kubelet-config\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.033835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033746 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-slash\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.033835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033781 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-cni-binary-copy\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.033835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033811 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-host\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.033835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033829 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-dbus\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033845 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033774 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033783 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9931cd13-1775-4f3d-9300-d2cb16df7acf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033896 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-service-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033911 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21966c11-af60-40d2-bde4-d6cb8296cad4-cni-binary-copy\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033919 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5abe8e87-10f7-453b-872f-ca366598dd92-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033949 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.033974 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-bin\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034003 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-tmp-dir\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.034098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034055 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034120 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034169 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034184 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-tmp-dir\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034202 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9931cd13-1775-4f3d-9300-d2cb16df7acf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034226 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-stats-auth\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034257 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034287 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-var-lib-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034314 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034346 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zwx\" (UniqueName: \"kubernetes.io/projected/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-api-access-g7zwx\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034418 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034452 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034481 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-config\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034417 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.034557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034508 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-multus\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034554 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlpj\" (UniqueName: \"kubernetes.io/projected/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-kube-api-access-bmlpj\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034575 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5abe8e87-10f7-453b-872f-ca366598dd92-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034602 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034618 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08beba50-863a-4f30-a1ad-324cfde2159f-service-ca-bundle\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034634 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034634 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9931cd13-1775-4f3d-9300-d2cb16df7acf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034673 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034713 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-systemd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034813 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034849 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034865 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-stats-auth\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034868 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034885 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034878 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-tuned\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.034981 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035018 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035042 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035078 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-cnibin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovnkube-config\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035107 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-netns\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035135 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-tmp-dir\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.035157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035145 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-tuned\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035190 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-tmp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035212 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2459def9-9723-4de1-9a93-5a2dfb7a3cca-agent-certs\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035228 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4142b823-6b14-49f2-bf14-e8231f9296f3-original-pull-secret\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035229 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rwq\" (UniqueName: \"kubernetes.io/projected/84eda726-2f51-4c10-a829-681fdbb559e2-kube-api-access-56rwq\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035264 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-hostroot\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035282 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035321 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035337 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dgl\" (UniqueName: \"kubernetes.io/projected/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe-kube-api-access-c4dgl\") pod \"kas-connection-checker-685895c7c9-kmzkx\" (UID: \"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe\") " pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035353 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035377 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-netns\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035409 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-log-socket\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035519 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbks\" (UniqueName: \"kubernetes.io/projected/b13507f7-c09f-46bd-b76b-50ba62300dfb-kube-api-access-tqbks\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035540 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035557 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-tmp-dir\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035583 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035618 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-tmp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035631 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkt5\" (UniqueName: \"kubernetes.io/projected/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-kube-api-access-bvkt5\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035717 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dj8\" (UniqueName: \"kubernetes.io/projected/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-kube-api-access-v2dj8\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035762 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-iptables-alerter-script\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035817 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dn2f\" (UniqueName: \"kubernetes.io/projected/3a8e67f5-3ce1-412e-afb4-efbae8b2db98-kube-api-access-6dn2f\") pod \"migrator-66df5d496b-dv4lg\" (UID: \"3a8e67f5-3ce1-412e-afb4-efbae8b2db98\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035843 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2459def9-9723-4de1-9a93-5a2dfb7a3cca-agent-certs\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035912 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-system-cni-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035929 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdf32f3-9268-4552-aa33-1b201380e88e-crio-socket\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.035943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035957 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7bn\" (UniqueName: \"kubernetes.io/projected/8a877cc0-db2f-42fc-9057-a62433401379-kube-api-access-7x7bn\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035973 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-run\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035974 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-iptables-alerter-script\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.035987 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036027 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036051 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-serving-cert\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036087 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-snapshots\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036139 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08beba50-863a-4f30-a1ad-324cfde2159f-snapshots\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036164 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036201 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-ovn\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036244 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-serving-cert\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036301 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysconfig\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036365 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036362 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95dba424-9ecd-43b9-a09e-8c643019e969-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036444 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-conf-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036461 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-conf\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036484 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036502 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjnp\" (UniqueName: \"kubernetes.io/projected/08beba50-863a-4f30-a1ad-324cfde2159f-kube-api-access-bwjnp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036523 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036537 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036540 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036578 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovn-node-metrics-cert\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036617 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-socket-dir-parent\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036649 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-var-lib-kubelet\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036679 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-device-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036681 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036704 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-systemd\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036745 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.036855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036857 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2459def9-9723-4de1-9a93-5a2dfb7a3cca-konnectivity-ca\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:22.037647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036891 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b13507f7-c09f-46bd-b76b-50ba62300dfb-ovn-node-metrics-cert\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.037647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.036994 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2459def9-9723-4de1-9a93-5a2dfb7a3cca-konnectivity-ca\") pod \"konnectivity-agent-hsqfc\" (UID: \"2459def9-9723-4de1-9a93-5a2dfb7a3cca\") " pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:22.037647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.037036 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.049497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.049476 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbcm\" (UniqueName: \"kubernetes.io/projected/21966c11-af60-40d2-bde4-d6cb8296cad4-kube-api-access-tgbcm\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.049661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.049626 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xhlv\" (UniqueName: \"kubernetes.io/projected/ee11341a-54bb-412f-ad81-2c8ef689247a-kube-api-access-2xhlv\") pod \"ingress-canary-5lcrb\" (UID: \"ee11341a-54bb-412f-ad81-2c8ef689247a\") " pod="openshift-ingress-canary/ingress-canary-5lcrb" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050413 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwc6h\" (UniqueName: \"kubernetes.io/projected/c837b6b5-8ffc-4265-8d5c-1cc96e97fe75-kube-api-access-zwc6h\") pod \"kas-connection-checker-685895c7c9-j9ct9\" (UID: \"c837b6b5-8ffc-4265-8d5c-1cc96e97fe75\") " pod="kube-system/kas-connection-checker-685895c7c9-j9ct9" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050430 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkscn\" (UniqueName: \"kubernetes.io/projected/d9781bb5-7481-4803-8ef9-2eca6954c446-kube-api-access-bkscn\") pod \"kas-connection-checker-685895c7c9-bnjl2\" (UID: \"d9781bb5-7481-4803-8ef9-2eca6954c446\") " pod="kube-system/kas-connection-checker-685895c7c9-bnjl2" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050647 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drzd\" (UniqueName: \"kubernetes.io/projected/9feefbd2-6be8-47ab-a427-e9e91ff48e1a-kube-api-access-6drzd\") pod \"migrator-66df5d496b-lxmgq\" (UID: \"9feefbd2-6be8-47ab-a427-e9e91ff48e1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxmgq" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050647 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnsn\" (UniqueName: \"kubernetes.io/projected/95dba424-9ecd-43b9-a09e-8c643019e969-kube-api-access-gcnsn\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050724 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050773 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f2m\" (UniqueName: \"kubernetes.io/projected/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-kube-api-access-l2f2m\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.050858 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw94c\" (UniqueName: \"kubernetes.io/projected/106e7cc6-df7c-4e5a-8876-1c0456a35023-kube-api-access-jw94c\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:22.051095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051030 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zh2\" (UniqueName: \"kubernetes.io/projected/7c7fd700-057c-4ceb-88c4-01620fa6329d-kube-api-access-q4zh2\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.051589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051254 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmlw\" (UniqueName: \"kubernetes.io/projected/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-kube-api-access-4wmlw\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.051589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051266 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwg9v\" (UniqueName: \"kubernetes.io/projected/9931cd13-1775-4f3d-9300-d2cb16df7acf-kube-api-access-mwg9v\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f\" (UID: \"9931cd13-1775-4f3d-9300-d2cb16df7acf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" Oct 30 21:09:22.051589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051257 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmf2d\" (UniqueName: \"kubernetes.io/projected/4f25c0ef-fd48-4bf6-b5a9-0261fe507a34-kube-api-access-wmf2d\") pod \"network-check-source-6f78d59f99-dq7ch\" (UID: \"4f25c0ef-fd48-4bf6-b5a9-0261fe507a34\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dq7ch" Oct 30 21:09:22.051589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051369 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jfvg\" (UniqueName: \"kubernetes.io/projected/2b8386a1-1b52-440d-b134-9c42390f70ba-kube-api-access-5jfvg\") pod \"volume-data-source-validator-58c9f5f46d-48pxc\" (UID: \"2b8386a1-1b52-440d-b134-9c42390f70ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-48pxc" Oct 30 21:09:22.051589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051527 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.051589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051569 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l656s\" (UniqueName: \"kubernetes.io/projected/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-kube-api-access-l656s\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.051874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.051747 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.052107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.052090 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cnl\" (UniqueName: \"kubernetes.io/projected/209912e2-5544-4a98-9282-ed7e28c449d0-kube-api-access-v4cnl\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.052259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.052241 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwjr\" (UniqueName: \"kubernetes.io/projected/5abe8e87-10f7-453b-872f-ca366598dd92-kube-api-access-7qwjr\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.052316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.052241 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8rm\" (UniqueName: \"kubernetes.io/projected/8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf-kube-api-access-gm8rm\") pod \"console-operator-7cd56b8dbc-gh745\" (UID: \"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:22.054140 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.054117 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlpj\" (UniqueName: \"kubernetes.io/projected/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-kube-api-access-bmlpj\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.054222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.054193 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") pod \"periodic-gathering-vb2td-csvl7\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:09:22.055330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.055284 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rwq\" (UniqueName: \"kubernetes.io/projected/84eda726-2f51-4c10-a829-681fdbb559e2-kube-api-access-56rwq\") pod \"service-ca-5fffc97bdf-hh45f\" (UID: \"84eda726-2f51-4c10-a829-681fdbb559e2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hh45f" Oct 30 21:09:22.057097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.056224 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.057097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.056296 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zwx\" (UniqueName: \"kubernetes.io/projected/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-api-access-g7zwx\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.057097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.056526 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dn2f\" (UniqueName: \"kubernetes.io/projected/3a8e67f5-3ce1-412e-afb4-efbae8b2db98-kube-api-access-6dn2f\") pod \"migrator-66df5d496b-dv4lg\" (UID: \"3a8e67f5-3ce1-412e-afb4-efbae8b2db98\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dv4lg" Oct 30 21:09:22.057097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.056790 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbks\" (UniqueName: \"kubernetes.io/projected/b13507f7-c09f-46bd-b76b-50ba62300dfb-kube-api-access-tqbks\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.057097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.056866 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dgl\" (UniqueName: \"kubernetes.io/projected/1ce09dc0-573a-4ac2-a6aa-6a37bead48fe-kube-api-access-c4dgl\") pod \"kas-connection-checker-685895c7c9-kmzkx\" (UID: \"1ce09dc0-573a-4ac2-a6aa-6a37bead48fe\") " pod="kube-system/kas-connection-checker-685895c7c9-kmzkx" Oct 30 21:09:22.057097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.057044 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncd4q\" (UniqueName: \"kubernetes.io/projected/0c1e1f33-a5b9-442c-8589-970667a86890-kube-api-access-ncd4q\") pod \"network-check-target-85c8w\" (UID: \"0c1e1f33-a5b9-442c-8589-970667a86890\") " pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:22.057421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.057302 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dj8\" (UniqueName: \"kubernetes.io/projected/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-kube-api-access-v2dj8\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.058038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.057619 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7bn\" (UniqueName: \"kubernetes.io/projected/8a877cc0-db2f-42fc-9057-a62433401379-kube-api-access-7x7bn\") pod \"service-ca-operator-544d749c8c-h4r5w\" (UID: \"8a877cc0-db2f-42fc-9057-a62433401379\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" Oct 30 21:09:22.058038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.057663 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkt5\" (UniqueName: \"kubernetes.io/projected/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-kube-api-access-bvkt5\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.058038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.057719 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjnp\" (UniqueName: \"kubernetes.io/projected/08beba50-863a-4f30-a1ad-324cfde2159f-kube-api-access-bwjnp\") pod \"insights-operator-7c4986d449-vws6h\" (UID: \"08beba50-863a-4f30-a1ad-324cfde2159f\") " pod="openshift-insights/insights-operator-7c4986d449-vws6h" Oct 30 21:09:22.138079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138050 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-etc-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138081 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-node-log\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138099 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-host-slash\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.138194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138114 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-kubelet\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.138194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138153 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-kubelet\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.138194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138164 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-node-log\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138172 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c-host-slash\") pod \"iptables-alerter-r6tmj\" (UID: \"4b62bf2e-5880-40f6-8d4b-ee60e2a8c30c\") " pod="openshift-network-operator/iptables-alerter-r6tmj" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138178 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-etc-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138182 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-hosts-file\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138220 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bafe23a-4bb7-477d-81a1-6024f5c3d9d4-hosts-file\") pod \"node-resolver-c7kgv\" (UID: \"9bafe23a-4bb7-477d-81a1-6024f5c3d9d4\") " pod="openshift-dns/node-resolver-c7kgv" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138228 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-kubelet-config\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138246 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-slash\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138260 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-host\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138273 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-dbus\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138289 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-bin\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138299 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-kubelet-config\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138309 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138327 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-slash\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138325 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e-host\") pod \"node-ca-87n87\" (UID: \"22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e\") " pod="openshift-image-registry/node-ca-87n87" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138332 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-var-lib-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138343 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-bin\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138364 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4142b823-6b14-49f2-bf14-e8231f9296f3-dbus\") pod \"global-pull-secret-syncer-b9l4r\" (UID: \"4142b823-6b14-49f2-bf14-e8231f9296f3\") " pod="kube-system/global-pull-secret-syncer-b9l4r" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138371 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138409 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-var-lib-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138428 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.138434 5039 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.138443 5039 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:09:22.138461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138452 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138479 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-multus\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.138507 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:22.638487342 +0000 UTC m=+1.848426970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : configmap "kube-rbac-proxy" not found Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138513 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-multus\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138530 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138541 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.138547 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs podName:7c7fd700-057c-4ceb-88c4-01620fa6329d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:22.6385299 +0000 UTC m=+1.848469526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs") pod "router-default-77b4b6cbf9-7xzk8" (UID: "7c7fd700-057c-4ceb-88c4-01620fa6329d") : secret "router-metrics-certs-default" not found Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138564 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138589 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-systemd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138626 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138651 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-cnibin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138686 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-netns\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138704 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-hostroot\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138702 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138726 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-cnibin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138684 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-systemd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138746 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-hostroot\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138757 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-netns\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138758 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138792 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-netns\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138815 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-log-socket\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138843 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-netns\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138846 5039 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138842 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-system-cni-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138871 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdf32f3-9268-4552-aa33-1b201380e88e-crio-socket\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138878 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-system-cni-dir\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138886 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-run\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138925 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138941 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-log-socket\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138948 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-run\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.138954 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139000 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-ovn\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139005 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdf32f3-9268-4552-aa33-1b201380e88e-crio-socket\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.139043 5039 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139058 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-ovn\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139084 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysconfig\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.139100 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:22.639086441 +0000 UTC m=+1.849026061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : secret "metrics-daemon-secret" not found Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139123 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysconfig\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139130 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-conf-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139156 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-conf\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139185 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139159 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-conf-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.139272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139224 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139253 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-socket-dir-parent\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139316 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-multus-socket-dir-parent\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139305 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-run-openvswitch\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139273 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-conf\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139336 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-var-lib-kubelet\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139378 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-device-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139410 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-var-lib-kubelet\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139427 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-systemd\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139456 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-device-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139455 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139499 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-systemd\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139522 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139598 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-sys-fs\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139895 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c7fd700-057c-4ceb-88c4-01620fa6329d-service-ca-bundle\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.139987 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-system-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140028 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140053 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-etc-kubernetes\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140057 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-system-cni-dir\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140094 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140118 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-modprobe-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140126 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-run-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140143 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-k8s-cni-cncf-io\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140201 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.140216 5039 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140229 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-lib-modules\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140257 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-cnibin\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140302 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95dba424-9ecd-43b9-a09e-8c643019e969-cnibin\") pod \"multus-additional-cni-plugins-cf9xp\" (UID: \"95dba424-9ecd-43b9-a09e-8c643019e969\") " pod="openshift-multus/multus-additional-cni-plugins-cf9xp" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140359 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140422 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-k8s-cni-cncf-io\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140478 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-lib-modules\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140533 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-modprobe-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140593 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-etc-kubernetes\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.140648 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:22.640253817 +0000 UTC m=+1.850193439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140703 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-bin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140728 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-kubelet\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140748 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-systemd-units\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140772 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140795 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140824 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140848 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140868 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-host\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140889 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-multus-certs\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140908 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-sys\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140934 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-netd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140954 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.140982 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-kubernetes\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141067 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-socket-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141085 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141112 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-kubelet\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141148 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-systemd-units\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.141151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141166 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/209912e2-5544-4a98-9282-ed7e28c449d0-registration-dir\") pod \"aws-ebs-csi-driver-node-xfsx4\" (UID: \"209912e2-5544-4a98-9282-ed7e28c449d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xfsx4" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141185 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141214 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-host\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141247 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-run-multus-certs\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141289 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-sys\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141295 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-sysctl-d\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141330 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b13507f7-c09f-46bd-b76b-50ba62300dfb-host-cni-netd\") pod \"ovnkube-node-psc8l\" (UID: \"b13507f7-c09f-46bd-b76b-50ba62300dfb\") " pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.141359 5039 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.141421 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:22.64138609 +0000 UTC m=+1.851325699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141466 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad5e0914-9ca0-4dbf-952e-1b9812908b1c-etc-kubernetes\") pod \"tuned-s586p\" (UID: \"ad5e0914-9ca0-4dbf-952e-1b9812908b1c\") " pod="openshift-cluster-node-tuning-operator/tuned-s586p" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141514 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5abe8e87-10f7-453b-872f-ca366598dd92-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xr594\" (UID: \"5abe8e87-10f7-453b-872f-ca366598dd92\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141630 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c5b111d-8074-4bda-8ac9-7d1f1f623c59-metrics-tls\") pod \"dns-default-rp8hx\" (UID: \"7c5b111d-8074-4bda-8ac9-7d1f1f623c59\") " pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141639 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21966c11-af60-40d2-bde4-d6cb8296cad4-host-var-lib-cni-bin\") pod \"multus-kzg7s\" (UID: \"21966c11-af60-40d2-bde4-d6cb8296cad4\") " pod="openshift-multus/multus-kzg7s" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.141928 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"image-registry-5b65c6946-57jbj\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.142867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.142099 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f71a2be6-1f45-43a6-ae97-8ebd007341fd-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-phllg\" (UID: \"f71a2be6-1f45-43a6-ae97-8ebd007341fd\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:22.143687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.143668 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"image-registry-66d5774b95-sq55b\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.206017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.205986 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.213564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.213545 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" Oct 30 21:09:22.214902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.214884 5039 scope.go:122] "RemoveContainer" containerID="5add8cbe7f653375cb09a2dc0a3552df37b40020b0d3dd25556e937c2e62a1eb" Oct 30 21:09:22.217990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.217971 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:22.218073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.218006 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" Oct 30 21:09:22.220793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.220774 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:22.228450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.228435 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:22.350470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.350379 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.490798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.490763 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.534231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.534148 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xr594"] Oct 30 21:09:22.537804 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:22.537773 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abe8e87_10f7_453b_872f_ca366598dd92.slice/crio-208066aded4d9320d9d0a634f82d13cab66e1739a28aea74428dd62a1a135f0d WatchSource:0}: Error finding container 208066aded4d9320d9d0a634f82d13cab66e1739a28aea74428dd62a1a135f0d: Status 404 returned error can't find the container with id 208066aded4d9320d9d0a634f82d13cab66e1739a28aea74428dd62a1a135f0d Oct 30 21:09:22.540039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.540024 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:09:22.571590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.571541 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:09:22.574433 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:22.574385 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57293ead_fcc2_47e7_8033_d0beec008c51.slice/crio-c840a0b20d58aabfca5ddc147ddc8f3c218f1cd19d6e320e92f4ac43aff10180 WatchSource:0}: Error finding container c840a0b20d58aabfca5ddc147ddc8f3c218f1cd19d6e320e92f4ac43aff10180: Status 404 returned error can't find the container with id c840a0b20d58aabfca5ddc147ddc8f3c218f1cd19d6e320e92f4ac43aff10180 Oct 30 21:09:22.607875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.607844 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rp8hx"] Oct 30 21:09:22.612227 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:22.612200 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5b111d_8074_4bda_8ac9_7d1f1f623c59.slice/crio-6a994ca8e7886c090921a5b8223746cb5f7462b1950d1c82846aac3be533d4d9 WatchSource:0}: Error finding container 6a994ca8e7886c090921a5b8223746cb5f7462b1950d1c82846aac3be533d4d9: Status 404 returned error can't find the container with id 6a994ca8e7886c090921a5b8223746cb5f7462b1950d1c82846aac3be533d4d9 Oct 30 21:09:22.637374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.637346 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-phllg"] Oct 30 21:09:22.640745 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:22.640713 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71a2be6_1f45_43a6_ae97_8ebd007341fd.slice/crio-09c7dcf7e5779bc19776a57c79af9b254c7a1f7bb5a66c6eeafd76ca746fd0ef WatchSource:0}: Error finding container 09c7dcf7e5779bc19776a57c79af9b254c7a1f7bb5a66c6eeafd76ca746fd0ef: Status 404 returned error can't find the container with id 09c7dcf7e5779bc19776a57c79af9b254c7a1f7bb5a66c6eeafd76ca746fd0ef Oct 30 21:09:22.646075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.646047 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.646147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.646103 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:22.646147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.646138 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.646233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.646158 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.646233 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.646179 5039 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:22.646233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.646202 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:22.646372 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.646240 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.646220972 +0000 UTC m=+2.856160581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:22.646372 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.646262 5039 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:22.646372 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.646274 5039 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:09:22.646372 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.646315 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.646299141 +0000 UTC m=+2.856238766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:22.646372 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:22.646335 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.646325716 +0000 UTC m=+2.856265329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : secret "metrics-daemon-secret" not found Oct 30 21:09:22.646801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.646782 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdf32f3-9268-4552-aa33-1b201380e88e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:22.648385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.648367 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7fd700-057c-4ceb-88c4-01620fa6329d-metrics-certs\") pod \"router-default-77b4b6cbf9-7xzk8\" (UID: \"7c7fd700-057c-4ceb-88c4-01620fa6329d\") " pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.669035 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.669011 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b65c6946-57jbj"] Oct 30 21:09:22.672347 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:22.672323 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccf83b0_6149_4f2b_b3b1_db339a5a245e.slice/crio-ab4ddc9cdcd069de542f4cfd3e52b5f4aeb0b7168005f44a76aaaa4a7bccedd6 WatchSource:0}: Error finding container ab4ddc9cdcd069de542f4cfd3e52b5f4aeb0b7168005f44a76aaaa4a7bccedd6: Status 404 returned error can't find the container with id ab4ddc9cdcd069de542f4cfd3e52b5f4aeb0b7168005f44a76aaaa4a7bccedd6 Oct 30 21:09:22.794842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.794780 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.812822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.812786 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:22.979711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.979650 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:09:22.980162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.979784 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" event={"ID":"8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf","Type":"ContainerStarted","Data":"d6c489579da137f1e4a9dd63050d7da1de7f9c7f913d0f3472062ad31b4fda45"} Oct 30 21:09:22.981417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.981376 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" event={"ID":"f71a2be6-1f45-43a6-ae97-8ebd007341fd","Type":"ContainerStarted","Data":"09c7dcf7e5779bc19776a57c79af9b254c7a1f7bb5a66c6eeafd76ca746fd0ef"} Oct 30 21:09:22.983274 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.983245 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" event={"ID":"57293ead-fcc2-47e7-8033-d0beec008c51","Type":"ContainerStarted","Data":"9754019f7ec6928416fe7ebb93d5f5eb8dc89313f8e184b25e73b8c231fe26fe"} Oct 30 21:09:22.983442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.983293 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" event={"ID":"57293ead-fcc2-47e7-8033-d0beec008c51","Type":"ContainerStarted","Data":"c840a0b20d58aabfca5ddc147ddc8f3c218f1cd19d6e320e92f4ac43aff10180"} Oct 30 21:09:22.985130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.985106 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" event={"ID":"dccf83b0-6149-4f2b-b3b1-db339a5a245e","Type":"ContainerStarted","Data":"cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db"} Oct 30 21:09:22.985243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.985148 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" event={"ID":"dccf83b0-6149-4f2b-b3b1-db339a5a245e","Type":"ContainerStarted","Data":"ab4ddc9cdcd069de542f4cfd3e52b5f4aeb0b7168005f44a76aaaa4a7bccedd6"} Oct 30 21:09:22.986558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.986525 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" event={"ID":"5abe8e87-10f7-453b-872f-ca366598dd92","Type":"ContainerStarted","Data":"208066aded4d9320d9d0a634f82d13cab66e1739a28aea74428dd62a1a135f0d"} Oct 30 21:09:22.987896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:22.987874 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rp8hx" event={"ID":"7c5b111d-8074-4bda-8ac9-7d1f1f623c59","Type":"ContainerStarted","Data":"6a994ca8e7886c090921a5b8223746cb5f7462b1950d1c82846aac3be533d4d9"} Oct 30 21:09:23.122223 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.122144 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77b4b6cbf9-7xzk8"] Oct 30 21:09:23.125720 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:23.125686 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7fd700_057c_4ceb_88c4_01620fa6329d.slice/crio-dc03bc86adc1c33881821f5025a255ef638b1f673de070cf4d894e9233d64023 WatchSource:0}: Error finding container dc03bc86adc1c33881821f5025a255ef638b1f673de070cf4d894e9233d64023: Status 404 returned error can't find the container with id dc03bc86adc1c33881821f5025a255ef638b1f673de070cf4d894e9233d64023 Oct 30 21:09:23.162839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.162803 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.353546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.353514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.419372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.419315 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hsqfc" Oct 30 21:09:23.495523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.495427 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.649123 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.649029 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.655016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.654985 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:23.655137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.655049 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:23.655137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.655100 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:23.655244 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:23.655222 5039 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:09:23.655350 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:23.655285 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls podName:dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.655267863 +0000 UTC m=+4.865207475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c8ptn" (UID: "dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5") : secret "samples-operator-tls" not found Oct 30 21:09:23.655536 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:23.655518 5039 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:23.655585 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:23.655533 5039 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:09:23.655585 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:23.655576 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.655562792 +0000 UTC m=+4.865502414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:23.655678 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:23.655601 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:25.655593459 +0000 UTC m=+4.865533067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : secret "metrics-daemon-secret" not found Oct 30 21:09:23.830937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.830868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.972367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.972326 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.997220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.996185 5039 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:23.997220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.996706 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" event={"ID":"7c7fd700-057c-4ceb-88c4-01620fa6329d","Type":"ContainerStarted","Data":"d5cb3d6d44c7c22f6d833a424b98a4b835bd8fcee21250370761ee895398b45d"} Oct 30 21:09:23.997220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.996754 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" event={"ID":"7c7fd700-057c-4ceb-88c4-01620fa6329d","Type":"ContainerStarted","Data":"dc03bc86adc1c33881821f5025a255ef638b1f673de070cf4d894e9233d64023"} Oct 30 21:09:23.997220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:23.996819 5039 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:24.111751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:24.111664 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.255654 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:24.255611 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.457544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:24.457501 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:24.528617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:24.528586 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:24.813852 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:24.813827 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:24.816535 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:24.816515 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:25.003185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.003089 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rp8hx" event={"ID":"7c5b111d-8074-4bda-8ac9-7d1f1f623c59","Type":"ContainerStarted","Data":"dc318aafa6b7fad53d83f015db40b2f2a33bc213e061ba6089ed728654b4ea92"} Oct 30 21:09:25.003185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.003153 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rp8hx" event={"ID":"7c5b111d-8074-4bda-8ac9-7d1f1f623c59","Type":"ContainerStarted","Data":"ade695ac16634f4c4be3fbb311e795e7e509c5b053bb76fd1d5dd4743ef44f8a"} Oct 30 21:09:25.003185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.003181 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:25.004744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.004715 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-phllg" event={"ID":"f71a2be6-1f45-43a6-ae97-8ebd007341fd","Type":"ContainerStarted","Data":"4bfa79c3621791901eae15f8b93b80df327945b4464b17e9c4b9a38d9330a861"} Oct 30 21:09:25.006371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.006339 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xr594" event={"ID":"5abe8e87-10f7-453b-872f-ca366598dd92","Type":"ContainerStarted","Data":"6450f8b33cd2c4374ae49b4eebed2a3eea864ababd5dd84092045361dfae7484"} Oct 30 21:09:25.265148 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.265109 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:25.401878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.401843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:25.554491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.554406 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:25.675479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.675437 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:25.675617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.675536 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:25.675617 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:25.675553 5039 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:09:25.675617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.675593 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:25.675721 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:25.675609 5039 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:09:25.675721 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:25.675617 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs podName:106e7cc6-df7c-4e5a-8876-1c0456a35023 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:29.675599429 +0000 UTC m=+8.885539051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs") pod "network-metrics-daemon-c88gd" (UID: "106e7cc6-df7c-4e5a-8876-1c0456a35023") : secret "metrics-daemon-secret" not found Oct 30 21:09:25.675721 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:25.675684 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls podName:ffdf32f3-9268-4552-aa33-1b201380e88e nodeName:}" failed. No retries permitted until 2026-10-30 21:09:29.675668381 +0000 UTC m=+8.885607989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq4d5" (UID: "ffdf32f3-9268-4552-aa33-1b201380e88e") : secret "insights-runtime-extractor-tls" not found Oct 30 21:09:25.677947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.677923 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c8ptn\" (UID: \"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:25.695050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.695021 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:25.813911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.813840 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" Oct 30 21:09:25.838206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:25.838180 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.000824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.000793 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.014365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.014336 5039 generic.go:356] "Generic (PLEG): container finished" podID="9931cd13-1775-4f3d-9300-d2cb16df7acf" containerID="8b9ceacf795cd1da5eac60fdf96b134a26ab19fa78b2185efe057b3e73a1281e" exitCode=0 Oct 30 21:09:26.014784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.014415 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" event={"ID":"9931cd13-1775-4f3d-9300-d2cb16df7acf","Type":"ContainerDied","Data":"8b9ceacf795cd1da5eac60fdf96b134a26ab19fa78b2185efe057b3e73a1281e"} Oct 30 21:09:26.014858 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.014839 5039 scope.go:122] "RemoveContainer" containerID="8b9ceacf795cd1da5eac60fdf96b134a26ab19fa78b2185efe057b3e73a1281e" Oct 30 21:09:26.016030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.016010 5039 generic.go:356] "Generic (PLEG): container finished" podID="08beba50-863a-4f30-a1ad-324cfde2159f" containerID="8762a741df60ec4b87cb543e44e5247c5256ab8f5180f5681ca879740c6a9d92" exitCode=0 Oct 30 21:09:26.016109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.016082 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vws6h" event={"ID":"08beba50-863a-4f30-a1ad-324cfde2159f","Type":"ContainerDied","Data":"8762a741df60ec4b87cb543e44e5247c5256ab8f5180f5681ca879740c6a9d92"} Oct 30 21:09:26.016215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.016171 5039 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:26.016627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.016609 5039 scope.go:122] "RemoveContainer" containerID="8762a741df60ec4b87cb543e44e5247c5256ab8f5180f5681ca879740c6a9d92" Oct 30 21:09:26.108200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.108162 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn"] Oct 30 21:09:26.265511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.265479 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.736501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:26.736469 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:27.022330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:27.022256 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-mzz4f" event={"ID":"9931cd13-1775-4f3d-9300-d2cb16df7acf","Type":"ContainerStarted","Data":"e2b7a65910d9eaca93da3f892839d3f99d2c15d22e8de4d5146a8e089160d6d8"} Oct 30 21:09:27.024141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:27.024116 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vws6h" event={"ID":"08beba50-863a-4f30-a1ad-324cfde2159f","Type":"ContainerStarted","Data":"82090a7f42aeadc53317eb2efc2404a7227675d16aa9f50f5d3cd76399529736"} Oct 30 21:09:27.025182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:27.025165 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" event={"ID":"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5","Type":"ContainerStarted","Data":"bc372c301815337cd506a148e48cb4c5abb33d78f58ffa1d22e6b0a17b78f100"} Oct 30 21:09:27.409230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:27.409193 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.601243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:27.601201 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.810895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:27.810852 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.007877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.007777 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.033676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.033646 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" event={"ID":"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5","Type":"ContainerStarted","Data":"a3bfc56f587486ab30d031d3fb2c977f06f4f3fa0508097e4c57b42da1c98316"} Oct 30 21:09:28.034041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.033718 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c8ptn" event={"ID":"dae60e05-bd8b-4ed8-9bf0-1ef3f12447d5","Type":"ContainerStarted","Data":"483266c3d067b305ae58490cf3947c3dc581ab08e6a6447001d9e3fbb04c7a8e"} Oct 30 21:09:28.168098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.168065 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.307516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.307434 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.597500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.597433 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:28.775167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.775135 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:28.775441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.775424 5039 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:28.806029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:28.805998 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:29.009642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.009610 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:29.181845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.181811 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:29.182248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.181963 5039 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:29.185385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.185367 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gh745" Oct 30 21:09:29.711699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.711664 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:29.711851 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.711713 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:29.713909 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.713881 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106e7cc6-df7c-4e5a-8876-1c0456a35023-metrics-certs\") pod \"network-metrics-daemon-c88gd\" (UID: \"106e7cc6-df7c-4e5a-8876-1c0456a35023\") " pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:29.714091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.714069 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdf32f3-9268-4552-aa33-1b201380e88e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq4d5\" (UID: \"ffdf32f3-9268-4552-aa33-1b201380e88e\") " pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:29.725940 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.725921 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c88gd" Oct 30 21:09:29.731501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:29.731485 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq4d5" Oct 30 21:09:30.022296 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:30.022228 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c88gd"] Oct 30 21:09:30.025370 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:30.025337 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106e7cc6_df7c_4e5a_8876_1c0456a35023.slice/crio-1707ceb9cb090e4b9c438621697295c298ea450df93a4b51ed955c94ef1a4c4e WatchSource:0}: Error finding container 1707ceb9cb090e4b9c438621697295c298ea450df93a4b51ed955c94ef1a4c4e: Status 404 returned error can't find the container with id 1707ceb9cb090e4b9c438621697295c298ea450df93a4b51ed955c94ef1a4c4e Oct 30 21:09:30.033054 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:30.032985 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sq4d5"] Oct 30 21:09:30.036285 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:30.036256 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdf32f3_9268_4552_aa33_1b201380e88e.slice/crio-055e0d430ea7e09b0994cf6db79748078edb8057bcf839b0535b154aabb895a8 WatchSource:0}: Error finding container 055e0d430ea7e09b0994cf6db79748078edb8057bcf839b0535b154aabb895a8: Status 404 returned error can't find the container with id 055e0d430ea7e09b0994cf6db79748078edb8057bcf839b0535b154aabb895a8 Oct 30 21:09:30.044993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:30.044964 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq4d5" event={"ID":"ffdf32f3-9268-4552-aa33-1b201380e88e","Type":"ContainerStarted","Data":"055e0d430ea7e09b0994cf6db79748078edb8057bcf839b0535b154aabb895a8"} Oct 30 21:09:30.046091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:30.046072 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c88gd" event={"ID":"106e7cc6-df7c-4e5a-8876-1c0456a35023","Type":"ContainerStarted","Data":"1707ceb9cb090e4b9c438621697295c298ea450df93a4b51ed955c94ef1a4c4e"} Oct 30 21:09:31.059641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:31.059603 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq4d5" event={"ID":"ffdf32f3-9268-4552-aa33-1b201380e88e","Type":"ContainerStarted","Data":"ab2c5a1aaa9dce4a61a83032e8e7666d231b64b7d12b9c10c525dc56bf30a21a"} Oct 30 21:09:31.060053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:31.059692 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq4d5" event={"ID":"ffdf32f3-9268-4552-aa33-1b201380e88e","Type":"ContainerStarted","Data":"dc83b1fe77a19b7d4c6587dbbd313fbd1b170a4e44c2674b401c3468e8e11c8a"} Oct 30 21:09:31.067670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:31.067641 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c88gd" event={"ID":"106e7cc6-df7c-4e5a-8876-1c0456a35023","Type":"ContainerStarted","Data":"92642014c37fa59fea5d23c890c033bf56265091302ba7c85ddc183bbbb4db70"} Oct 30 21:09:32.074246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:32.074203 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c88gd" event={"ID":"106e7cc6-df7c-4e5a-8876-1c0456a35023","Type":"ContainerStarted","Data":"946c7318c877826f71db90e34fc8e1d89b8c3a65505a4b43ee13726b45854506"} Oct 30 21:09:32.221739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:32.221713 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:32.221852 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:32.221828 5039 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:09:32.222948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:32.222928 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77b4b6cbf9-7xzk8" Oct 30 21:09:32.733763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:32.733732 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psc8l" Oct 30 21:09:33.083121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:33.083026 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq4d5" event={"ID":"ffdf32f3-9268-4552-aa33-1b201380e88e","Type":"ContainerStarted","Data":"595d9b72e30d5fad94e7fdf3de1aaf12511984e8997adbe63ab949c98bd14e71"} Oct 30 21:09:34.300376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:34.300339 5039 patch_prober.go:29] interesting pod/image-registry-66d5774b95-sq55b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:09:34.300376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:34.300420 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" podUID="57293ead-fcc2-47e7-8033-d0beec008c51" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:09:35.018721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.018669 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rp8hx" Oct 30 21:09:35.023935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.023906 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6b6wf-s5ppz"] Oct 30 21:09:35.030040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.030013 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.051243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.051203 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg"] Oct 30 21:09:35.054851 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.054823 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:35.071473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.071447 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6b6wf-s5ppz"] Oct 30 21:09:35.076096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.076072 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:09:35.090435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.090278 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:09:35.092209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.092152 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nc59x\"" Oct 30 21:09:35.107310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.107283 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg"] Oct 30 21:09:35.150343 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.150314 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-tsqlg"] Oct 30 21:09:35.153993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.153968 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.160127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.160100 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.160295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.160265 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.160439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.160423 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bzw\" (UniqueName: \"kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.160510 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.160458 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2586dac-e3ce-4d40-9ae1-52dacdd9acee-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h46wg\" (UID: \"b2586dac-e3ce-4d40-9ae1-52dacdd9acee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:35.160615 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.160597 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.167875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.167852 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:09:35.167968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.167937 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:09:35.181178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.180420 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tsqlg"] Oct 30 21:09:35.187185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.187164 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wmkg6\"" Oct 30 21:09:35.238586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.238558 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f6cf8465-hcpb2"] Oct 30 21:09:35.242358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.242322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:35.242513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.242431 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.262560 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baac63f8-6718-46c9-a1de-2a382cf188c9-tmp\") pod \"downloads-6dd789c8b7-tsqlg\" (UID: \"baac63f8-6718-46c9-a1de-2a382cf188c9\") " pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.262628 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bzw\" (UniqueName: \"kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.262667 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2586dac-e3ce-4d40-9ae1-52dacdd9acee-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h46wg\" (UID: \"b2586dac-e3ce-4d40-9ae1-52dacdd9acee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.262713 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.262754 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.263424 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6dv\" (UniqueName: \"kubernetes.io/projected/baac63f8-6718-46c9-a1de-2a382cf188c9-kube-api-access-kg6dv\") pod \"downloads-6dd789c8b7-tsqlg\" (UID: \"baac63f8-6718-46c9-a1de-2a382cf188c9\") " pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.263818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.263534 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.269119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.265126 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.269119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.265463 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.269119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.266838 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.269119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.267921 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2586dac-e3ce-4d40-9ae1-52dacdd9acee-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-h46wg\" (UID: \"b2586dac-e3ce-4d40-9ae1-52dacdd9acee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:35.277032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.277004 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f6cf8465-hcpb2"] Oct 30 21:09:35.287275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.287249 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bzw\" (UniqueName: \"kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw\") pod \"periodic-gathering-6b6wf-s5ppz\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.342515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.342485 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:09:35.364978 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.364946 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-registry-certificates\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.364999 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6dv\" (UniqueName: \"kubernetes.io/projected/baac63f8-6718-46c9-a1de-2a382cf188c9-kube-api-access-kg6dv\") pod \"downloads-6dd789c8b7-tsqlg\" (UID: \"baac63f8-6718-46c9-a1de-2a382cf188c9\") " pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.365122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365098 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-installation-pull-secrets\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365142 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-ca-trust-extracted\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365197 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-registry-tls\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365309 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-bound-sa-token\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365369 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baac63f8-6718-46c9-a1de-2a382cf188c9-tmp\") pod \"downloads-6dd789c8b7-tsqlg\" (UID: \"baac63f8-6718-46c9-a1de-2a382cf188c9\") " pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.365467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365447 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-image-registry-private-configuration\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365594 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-trusted-ca\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365631 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tnq\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-kube-api-access-f7tnq\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.365890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.365853 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baac63f8-6718-46c9-a1de-2a382cf188c9-tmp\") pod \"downloads-6dd789c8b7-tsqlg\" (UID: \"baac63f8-6718-46c9-a1de-2a382cf188c9\") " pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.368127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.368107 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:35.388955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.388927 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6dv\" (UniqueName: \"kubernetes.io/projected/baac63f8-6718-46c9-a1de-2a382cf188c9-kube-api-access-kg6dv\") pod \"downloads-6dd789c8b7-tsqlg\" (UID: \"baac63f8-6718-46c9-a1de-2a382cf188c9\") " pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466487 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tnq\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-kube-api-access-f7tnq\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466559 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-registry-certificates\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466603 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-installation-pull-secrets\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466633 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-ca-trust-extracted\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466681 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-registry-tls\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466711 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-bound-sa-token\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466773 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-image-registry-private-configuration\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.466802 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-trusted-ca\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.467649 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.467863 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-trusted-ca\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.467866 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-ca-trust-extracted\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.468971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.468755 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-registry-certificates\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.470814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.470784 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-installation-pull-secrets\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.470814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.470791 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-image-registry-private-configuration\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.471588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.471560 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-registry-tls\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.483935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.483825 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-bound-sa-token\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.483935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.483903 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tnq\" (UniqueName: \"kubernetes.io/projected/951cac7b-c383-4a73-b1cb-5fce1ff8a4f4-kube-api-access-f7tnq\") pod \"image-registry-f6cf8465-hcpb2\" (UID: \"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4\") " pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.556014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.555738 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:35.647197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.644597 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6b6wf-s5ppz"] Oct 30 21:09:35.682203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.682179 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg"] Oct 30 21:09:35.767598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.765844 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tsqlg"] Oct 30 21:09:35.857113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:35.857032 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f6cf8465-hcpb2"] Oct 30 21:09:35.859760 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:35.859728 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951cac7b_c383_4a73_b1cb_5fce1ff8a4f4.slice/crio-1123da8f344d31282b6aa19052f1e3a1d3ece26961f5f465edc40e34f8aadee8 WatchSource:0}: Error finding container 1123da8f344d31282b6aa19052f1e3a1d3ece26961f5f465edc40e34f8aadee8: Status 404 returned error can't find the container with id 1123da8f344d31282b6aa19052f1e3a1d3ece26961f5f465edc40e34f8aadee8 Oct 30 21:09:36.109490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.109322 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tsqlg" event={"ID":"baac63f8-6718-46c9-a1de-2a382cf188c9","Type":"ContainerStarted","Data":"09c4949136b7a668c8ce5adfe2a5a10b81336c3418be79010eac8094f8dcdb49"} Oct 30 21:09:36.111836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.111766 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" event={"ID":"b2586dac-e3ce-4d40-9ae1-52dacdd9acee","Type":"ContainerStarted","Data":"3cd48225b7d343285e0a9f060576941e80b344d1ff1e5da5882a22df31ccfc70"} Oct 30 21:09:36.114690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.113856 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" event={"ID":"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4","Type":"ContainerStarted","Data":"041ec98e6adbb98cc652f59fcc5fdbf45223a62a82db52ea644d6653f11c946a"} Oct 30 21:09:36.114690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.113900 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" event={"ID":"951cac7b-c383-4a73-b1cb-5fce1ff8a4f4","Type":"ContainerStarted","Data":"1123da8f344d31282b6aa19052f1e3a1d3ece26961f5f465edc40e34f8aadee8"} Oct 30 21:09:36.114690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.114482 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:36.117097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.117066 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" event={"ID":"ea35dc79-a590-4921-8efb-9e2084a5a401","Type":"ContainerStarted","Data":"6c3d55783e3ee8336e7e0ad6946b84fd5e21905b9048ea92962933b2b953f584"} Oct 30 21:09:36.117238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.117103 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" event={"ID":"ea35dc79-a590-4921-8efb-9e2084a5a401","Type":"ContainerStarted","Data":"3dd78e1e62741e5192e19740e1195767807b985a891c10fee94983d05e0688d8"} Oct 30 21:09:36.154655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.154580 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" podStartSLOduration=1.154560733 podStartE2EDuration="1.154560733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:36.153552204 +0000 UTC m=+15.363491831" watchObservedRunningTime="2026-10-30 21:09:36.154560733 +0000 UTC m=+15.364500365" Oct 30 21:09:36.713544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.713516 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rp8hx_7c5b111d-8074-4bda-8ac9-7d1f1f623c59/dns/0.log" Oct 30 21:09:36.900412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:36.899430 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rp8hx_7c5b111d-8074-4bda-8ac9-7d1f1f623c59/kube-rbac-proxy/0.log" Oct 30 21:09:37.698602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:37.698569 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c7kgv_9bafe23a-4bb7-477d-81a1-6024f5c3d9d4/dns-node-resolver/0.log" Oct 30 21:09:37.867222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:37.867182 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:38.104594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.104511 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b65c6946-57jbj_dccf83b0-6149-4f2b-b3b1-db339a5a245e/registry/0.log" Oct 30 21:09:38.132647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.132459 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" event={"ID":"b2586dac-e3ce-4d40-9ae1-52dacdd9acee","Type":"ContainerStarted","Data":"845e989aad9f61acdf7a34333cc8419e9659f34d72eb3cb125fbd10d17f06a47"} Oct 30 21:09:38.132811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.132753 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:38.137724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.137683 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" Oct 30 21:09:38.165669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.165617 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" podStartSLOduration=4.165604174 podStartE2EDuration="4.165604174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:36.186916813 +0000 UTC m=+15.396856438" watchObservedRunningTime="2026-10-30 21:09:38.165604174 +0000 UTC m=+17.375543798" Oct 30 21:09:38.165958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.165931 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-h46wg" podStartSLOduration=1.806590315 podStartE2EDuration="3.165926516s" totalImagesPullingTime="1.359336201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:38.165267956 +0000 UTC m=+17.375207585" watchObservedRunningTime="2026-10-30 21:09:38.165926516 +0000 UTC m=+17.375866174" Oct 30 21:09:38.299361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.299321 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66d5774b95-sq55b_57293ead-fcc2-47e7-8033-d0beec008c51/registry/0.log" Oct 30 21:09:38.498223 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.498191 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f6cf8465-hcpb2_951cac7b-c383-4a73-b1cb-5fce1ff8a4f4/registry/0.log" Oct 30 21:09:38.696940 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.696908 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-87n87_22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e/node-ca/0.log" Oct 30 21:09:38.757592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.757515 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:38.912160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:38.912116 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:39.064781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:39.064699 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:39.297304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:39.297273 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b4b6cbf9-7xzk8_7c7fd700-057c-4ceb-88c4-01620fa6329d/router/0.log" Oct 30 21:09:39.498539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:39.498505 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5lcrb_ee11341a-54bb-412f-ad81-2c8ef689247a/serve-healthcheck-canary/0.log" Oct 30 21:09:40.552212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:40.552170 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:40.720477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:40.720444 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:41.526838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:41.526804 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:42.123576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.123539 5039 patch_prober.go:29] interesting pod/image-registry-5b65c6946-57jbj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:09:42.124013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.123600 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" podUID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:09:42.245930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.245815 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:42.451928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.451871 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:42.509688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.509561 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fbmmk"] Oct 30 21:09:42.517004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.516979 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.533253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.533212 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:09:42.533525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.533237 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28pwt\"" Oct 30 21:09:42.542326 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.542303 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:09:42.548461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548432 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47k42\" (UniqueName: \"kubernetes.io/projected/a212b647-d67a-4fa5-a9a9-328d04378aea-kube-api-access-47k42\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548487 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-tls\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548513 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548538 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-wtmp\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548572 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548606 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-sys\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548635 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-textfile\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548660 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-root\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.548705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.548681 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a212b647-d67a-4fa5-a9a9-328d04378aea-metrics-client-ca\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.576002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.575922 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:09:42.576002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.575970 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:09:42.604383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.604353 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:42.649035 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649000 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47k42\" (UniqueName: \"kubernetes.io/projected/a212b647-d67a-4fa5-a9a9-328d04378aea-kube-api-access-47k42\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649062 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-tls\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649091 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649121 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-wtmp\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649196 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649248 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-sys\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649272 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-wtmp\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649281 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-textfile\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649308 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-root\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649333 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a212b647-d67a-4fa5-a9a9-328d04378aea-metrics-client-ca\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649375 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-sys\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649482 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a212b647-d67a-4fa5-a9a9-328d04378aea-root\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649673 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-textfile\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.649960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649933 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.650037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.649984 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a212b647-d67a-4fa5-a9a9-328d04378aea-metrics-client-ca\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.651830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.651809 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.651914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.651823 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a212b647-d67a-4fa5-a9a9-328d04378aea-node-exporter-tls\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.689022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.688989 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47k42\" (UniqueName: \"kubernetes.io/projected/a212b647-d67a-4fa5-a9a9-328d04378aea-kube-api-access-47k42\") pod \"node-exporter-fbmmk\" (UID: \"a212b647-d67a-4fa5-a9a9-328d04378aea\") " pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.769113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.769033 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:42.845012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.844973 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbmmk" Oct 30 21:09:42.860177 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:09:42.859591 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda212b647_d67a_4fa5_a9a9_328d04378aea.slice/crio-08e3b400b87390d6b7cca605a9798e3dc57389ce29b02873ec21b9669a2a9657 WatchSource:0}: Error finding container 08e3b400b87390d6b7cca605a9798e3dc57389ce29b02873ec21b9669a2a9657: Status 404 returned error can't find the container with id 08e3b400b87390d6b7cca605a9798e3dc57389ce29b02873ec21b9669a2a9657 Oct 30 21:09:42.923003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:42.922961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:43.142772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:43.142037 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:43.156662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:43.156623 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbmmk" event={"ID":"a212b647-d67a-4fa5-a9a9-328d04378aea","Type":"ContainerStarted","Data":"08e3b400b87390d6b7cca605a9798e3dc57389ce29b02873ec21b9669a2a9657"} Oct 30 21:09:43.709639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:43.709601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:43.867970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:43.867930 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:44.069121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:44.069091 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:44.162941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:44.162907 5039 generic.go:356] "Generic (PLEG): container finished" podID="a212b647-d67a-4fa5-a9a9-328d04378aea" containerID="03693b35995679bac85188559eddbec7ae0eb3684e50ec42fe46b1c0d5710515" exitCode=0 Oct 30 21:09:44.163352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:44.162950 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbmmk" event={"ID":"a212b647-d67a-4fa5-a9a9-328d04378aea","Type":"ContainerDied","Data":"03693b35995679bac85188559eddbec7ae0eb3684e50ec42fe46b1c0d5710515"} Oct 30 21:09:44.397770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:44.397730 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:44.598876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:44.598801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:45.011510 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.011480 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:09:45.082086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.082059 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:09:45.171485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.171446 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbmmk" event={"ID":"a212b647-d67a-4fa5-a9a9-328d04378aea","Type":"ContainerStarted","Data":"fc1934daf2c97181dea42a1705eebef45d523c9b6415c94f9be61650a6371fd1"} Oct 30 21:09:45.171931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.171507 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbmmk" event={"ID":"a212b647-d67a-4fa5-a9a9-328d04378aea","Type":"ContainerStarted","Data":"1ff84063b09b123b8515534f6d7b256a7e28792454269dab269fbd6904ab7ad9"} Oct 30 21:09:45.210735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.210680 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fbmmk" podStartSLOduration=2.069887681 podStartE2EDuration="3.210665424s" totalImagesPullingTime="1.064699018s" totalInitContainerRuntime="76.078725ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:45.209595793 +0000 UTC m=+24.419535436" watchObservedRunningTime="2026-10-30 21:09:45.210665424 +0000 UTC m=+24.420605051" Oct 30 21:09:45.394334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.394281 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:45.536286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.536248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:45.768006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.767912 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:45.912194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:45.912146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:46.091999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:46.091906 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:46.521732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:46.521688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:46.660137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:46.660091 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:46.739867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:46.739830 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-85c8w" Oct 30 21:09:47.350786 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:47.350736 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:47.502644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:47.502607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:47.695790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:47.695742 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:49.462159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:49.462123 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:50.661855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:50.661821 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:50.796295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:50.796248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:51.122079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:51.121988 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:51.275727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:51.275673 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:52.928449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:52.928269 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:53.216246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:53.216151 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tsqlg" event={"ID":"baac63f8-6718-46c9-a1de-2a382cf188c9","Type":"ContainerStarted","Data":"9777ea8cadac3befb7a1ea65e3744c33c6f2067a59d3f44e4dfcdd4ef316214a"} Oct 30 21:09:53.216443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:53.216361 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:53.238705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:53.238672 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-tsqlg" Oct 30 21:09:53.269239 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:53.269179 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-tsqlg" podStartSLOduration=1.7772976329999999 podStartE2EDuration="18.269161811s" totalImagesPullingTime="16.491864178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:53.25952092 +0000 UTC m=+32.469460549" watchObservedRunningTime="2026-10-30 21:09:53.269161811 +0000 UTC m=+32.479101448" Oct 30 21:09:55.192175 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:09:55.192129 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:09:58.145512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:58.145482 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f6cf8465-hcpb2" Oct 30 21:09:58.232565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:58.232302 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b65c6946-57jbj"] Oct 30 21:09:58.441545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:58.440554 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:58.577620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:58.577580 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:59.736344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:59.736308 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:59.885998 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:09:59.885969 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:00.101906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.101813 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" podUID="57293ead-fcc2-47e7-8033-d0beec008c51" containerName="registry" containerID="cri-o://9754019f7ec6928416fe7ebb93d5f5eb8dc89313f8e184b25e73b8c231fe26fe" gracePeriod=30 Oct 30 21:10:00.262490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.262448 5039 generic.go:356] "Generic (PLEG): container finished" podID="57293ead-fcc2-47e7-8033-d0beec008c51" containerID="9754019f7ec6928416fe7ebb93d5f5eb8dc89313f8e184b25e73b8c231fe26fe" exitCode=0 Oct 30 21:10:00.262631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.262610 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" event={"ID":"57293ead-fcc2-47e7-8033-d0beec008c51","Type":"ContainerDied","Data":"9754019f7ec6928416fe7ebb93d5f5eb8dc89313f8e184b25e73b8c231fe26fe"} Oct 30 21:10:00.407763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.407735 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:10:00.427856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.427817 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.427999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.427886 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.427999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.427927 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.428114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.428019 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.428114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.428047 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.428114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.428083 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.428245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.428120 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.428245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.428151 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") pod \"57293ead-fcc2-47e7-8033-d0beec008c51\" (UID: \"57293ead-fcc2-47e7-8033-d0beec008c51\") " Oct 30 21:10:00.429069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.429035 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:00.429226 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.429196 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:00.431851 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.431792 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:00.431975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.431949 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:00.432304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.432277 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:00.433609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.433582 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "kube-api-access-bsgr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:00.433704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.433684 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:00.437036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.436999 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted" pod "57293ead-fcc2-47e7-8033-d0beec008c51" (UID: "57293ead-fcc2-47e7-8033-d0beec008c51"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:00.529620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529586 5039 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-bound-sa-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529617 5039 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-registry-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529627 5039 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-registry-certificates\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529636 5039 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57293ead-fcc2-47e7-8033-d0beec008c51-trusted-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529645 5039 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57293ead-fcc2-47e7-8033-d0beec008c51-ca-trust-extracted\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529654 5039 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-image-registry-private-configuration\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529664 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsgr4\" (UniqueName: \"kubernetes.io/projected/57293ead-fcc2-47e7-8033-d0beec008c51-kube-api-access-bsgr4\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:00.529847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:00.529672 5039 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57293ead-fcc2-47e7-8033-d0beec008c51-installation-pull-secrets\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:01.267957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.267867 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" event={"ID":"57293ead-fcc2-47e7-8033-d0beec008c51","Type":"ContainerDied","Data":"c840a0b20d58aabfca5ddc147ddc8f3c218f1cd19d6e320e92f4ac43aff10180"} Oct 30 21:10:01.267957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.267910 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5774b95-sq55b" Oct 30 21:10:01.268509 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.267961 5039 scope.go:122] "RemoveContainer" containerID="9754019f7ec6928416fe7ebb93d5f5eb8dc89313f8e184b25e73b8c231fe26fe" Oct 30 21:10:01.326923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.326893 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:10:01.336461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.336436 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66d5774b95-sq55b"] Oct 30 21:10:01.539083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.538974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:01.728855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.728817 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:01.876416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:01.876316 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:02.211749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:02.211718 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:02.363698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:02.363661 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:02.924059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:02.924025 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57293ead-fcc2-47e7-8033-d0beec008c51" path="/var/lib/kubelet/pods/57293ead-fcc2-47e7-8033-d0beec008c51/volumes" Oct 30 21:10:05.665123 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:05.665087 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:05.803237 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:05.803198 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:06.558788 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.558749 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vb2td-csvl7"] Oct 30 21:10:06.559368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.559334 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vb2td-csvl7" podUID="d887faa8-91ab-4a8e-8b92-38a199ef7081" containerName="insights-gathering" containerID="cri-o://c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296" gracePeriod=30 Oct 30 21:10:06.849351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.849324 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:10:06.886122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886092 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") pod \"d887faa8-91ab-4a8e-8b92-38a199ef7081\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " Oct 30 21:10:06.886278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886158 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") pod \"d887faa8-91ab-4a8e-8b92-38a199ef7081\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " Oct 30 21:10:06.886278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886250 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") pod \"d887faa8-91ab-4a8e-8b92-38a199ef7081\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " Oct 30 21:10:06.886418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886318 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") pod \"d887faa8-91ab-4a8e-8b92-38a199ef7081\" (UID: \"d887faa8-91ab-4a8e-8b92-38a199ef7081\") " Oct 30 21:10:06.886564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886542 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle" pod "d887faa8-91ab-4a8e-8b92-38a199ef7081" (UID: "d887faa8-91ab-4a8e-8b92-38a199ef7081"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:06.886651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886585 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle" pod "d887faa8-91ab-4a8e-8b92-38a199ef7081" (UID: "d887faa8-91ab-4a8e-8b92-38a199ef7081"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:06.886651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.886616 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path" pod "d887faa8-91ab-4a8e-8b92-38a199ef7081" (UID: "d887faa8-91ab-4a8e-8b92-38a199ef7081"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:06.888711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.888687 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn" pod "d887faa8-91ab-4a8e-8b92-38a199ef7081" (UID: "d887faa8-91ab-4a8e-8b92-38a199ef7081"). InnerVolumeSpecName "kube-api-access-6zhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:06.987078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.987040 5039 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d887faa8-91ab-4a8e-8b92-38a199ef7081-archives-path\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:06.987078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.987071 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zhcn\" (UniqueName: \"kubernetes.io/projected/d887faa8-91ab-4a8e-8b92-38a199ef7081-kube-api-access-6zhcn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:06.987078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.987081 5039 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-trusted-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:06.987277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:06.987090 5039 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d887faa8-91ab-4a8e-8b92-38a199ef7081-service-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:07.294840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.294806 5039 generic.go:356] "Generic (PLEG): container finished" podID="d887faa8-91ab-4a8e-8b92-38a199ef7081" containerID="c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296" exitCode=2 Oct 30 21:10:07.295014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.294852 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vb2td-csvl7" Oct 30 21:10:07.295014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.294876 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vb2td-csvl7" event={"ID":"d887faa8-91ab-4a8e-8b92-38a199ef7081","Type":"ContainerDied","Data":"c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296"} Oct 30 21:10:07.295014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.294916 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vb2td-csvl7" event={"ID":"d887faa8-91ab-4a8e-8b92-38a199ef7081","Type":"ContainerDied","Data":"b05f5e8f5b318a108df1b249626944c984254fb1ace66e597e6076b833a29a46"} Oct 30 21:10:07.295014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.294935 5039 scope.go:122] "RemoveContainer" containerID="c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296" Oct 30 21:10:07.303345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.303328 5039 scope.go:122] "RemoveContainer" containerID="c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296" Oct 30 21:10:07.303637 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:10:07.303617 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296\": container with ID starting with c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296 not found: ID does not exist" containerID="c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296" Oct 30 21:10:07.303705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.303645 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296"} err="failed to get container status \"c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296\": rpc error: code = NotFound desc = could not find container \"c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296\": container with ID starting with c54aa6f6e01f08a77f57dd22cfb1842e87d93b1ff9a360a906243ff3a03c4296 not found: ID does not exist" Oct 30 21:10:07.340432 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.340407 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vb2td-csvl7"] Oct 30 21:10:07.349643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:07.349615 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vb2td-csvl7"] Oct 30 21:10:08.919024 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:08.918989 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d887faa8-91ab-4a8e-8b92-38a199ef7081" path="/var/lib/kubelet/pods/d887faa8-91ab-4a8e-8b92-38a199ef7081/volumes" Oct 30 21:10:10.235588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:10.235553 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:10.403842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:10.403810 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:19.243555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:19.243514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:19.399895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:19.399849 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:20.335840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:20.335790 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:20.663187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:20.663151 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:20.811156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:20.811120 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.232486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:21.232421 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.368310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:21.368276 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.797204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:21.797173 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.954848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:21.954808 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:23.259687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.259626 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" podUID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" containerName="registry" containerID="cri-o://cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db" gracePeriod=30 Oct 30 21:10:23.529625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.529570 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:10:23.635861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.635832 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.635878 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.635909 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.635946 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.635975 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.636020 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.636083 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.636113 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") pod \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\" (UID: \"dccf83b0-6149-4f2b-b3b1-db339a5a245e\") " Oct 30 21:10:23.636365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.636244 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:23.636596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.636380 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:23.636596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.636424 5039 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-certificates\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.638624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.638596 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:23.638785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.638752 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:23.638893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.638874 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "kube-api-access-jwk4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:23.638985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.638962 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:23.639033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.639020 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:23.642588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.642565 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted" pod "dccf83b0-6149-4f2b-b3b1-db339a5a245e" (UID: "dccf83b0-6149-4f2b-b3b1-db339a5a245e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:23.737222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737194 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwk4q\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-kube-api-access-jwk4q\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.737222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737217 5039 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-installation-pull-secrets\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.737384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737228 5039 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dccf83b0-6149-4f2b-b3b1-db339a5a245e-trusted-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.737384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737240 5039 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dccf83b0-6149-4f2b-b3b1-db339a5a245e-image-registry-private-configuration\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.737384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737249 5039 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-registry-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.737384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737258 5039 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dccf83b0-6149-4f2b-b3b1-db339a5a245e-bound-sa-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:23.737384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:23.737266 5039 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dccf83b0-6149-4f2b-b3b1-db339a5a245e-ca-trust-extracted\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:10:24.377373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.377340 5039 generic.go:356] "Generic (PLEG): container finished" podID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" containerID="cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db" exitCode=0 Oct 30 21:10:24.377861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.377407 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" Oct 30 21:10:24.377861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.377424 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" event={"ID":"dccf83b0-6149-4f2b-b3b1-db339a5a245e","Type":"ContainerDied","Data":"cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db"} Oct 30 21:10:24.377861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.377482 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b65c6946-57jbj" event={"ID":"dccf83b0-6149-4f2b-b3b1-db339a5a245e","Type":"ContainerDied","Data":"ab4ddc9cdcd069de542f4cfd3e52b5f4aeb0b7168005f44a76aaaa4a7bccedd6"} Oct 30 21:10:24.377861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.377508 5039 scope.go:122] "RemoveContainer" containerID="cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db" Oct 30 21:10:24.385992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.385972 5039 scope.go:122] "RemoveContainer" containerID="cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db" Oct 30 21:10:24.386242 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:10:24.386223 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db\": container with ID starting with cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db not found: ID does not exist" containerID="cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db" Oct 30 21:10:24.386309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.386250 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db"} err="failed to get container status \"cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db\": rpc error: code = NotFound desc = could not find container \"cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db\": container with ID starting with cafddc9108ec98ab1d143f04d41fb111fc27c0f6d96459029c6f2a428f9e50db not found: ID does not exist" Oct 30 21:10:24.429958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.429935 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b65c6946-57jbj"] Oct 30 21:10:24.439743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.439714 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b65c6946-57jbj"] Oct 30 21:10:24.919603 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:24.919570 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" path="/var/lib/kubelet/pods/dccf83b0-6149-4f2b-b3b1-db339a5a245e/volumes" Oct 30 21:10:25.184529 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:10:25.184453 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:10:26.391710 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:26.391664 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:26.538874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:26.538837 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:30.404147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:30.404113 5039 generic.go:356] "Generic (PLEG): container finished" podID="8a877cc0-db2f-42fc-9057-a62433401379" containerID="25f03af7906dbea31e9d84290487d8f67ca114773e579c8d47728e6266789f0e" exitCode=0 Oct 30 21:10:30.404546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:30.404187 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" event={"ID":"8a877cc0-db2f-42fc-9057-a62433401379","Type":"ContainerDied","Data":"25f03af7906dbea31e9d84290487d8f67ca114773e579c8d47728e6266789f0e"} Oct 30 21:10:30.404589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:30.404566 5039 scope.go:122] "RemoveContainer" containerID="25f03af7906dbea31e9d84290487d8f67ca114773e579c8d47728e6266789f0e" Oct 30 21:10:31.409724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:31.409691 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-h4r5w" event={"ID":"8a877cc0-db2f-42fc-9057-a62433401379","Type":"ContainerStarted","Data":"9caa3c80836e06fa713ed809fb9aa75102c7bfb945cdf326c932076ed2eaea07"} Oct 30 21:10:32.700453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:32.700411 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:32.849591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:32.849544 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:35.661274 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:35.661239 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:35.800762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:10:35.800726 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:55.184583 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:10:55.184536 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:11:01.535426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:01.535376 5039 generic.go:356] "Generic (PLEG): container finished" podID="ea35dc79-a590-4921-8efb-9e2084a5a401" containerID="6c3d55783e3ee8336e7e0ad6946b84fd5e21905b9048ea92962933b2b953f584" exitCode=0 Oct 30 21:11:01.535801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:01.535451 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" event={"ID":"ea35dc79-a590-4921-8efb-9e2084a5a401","Type":"ContainerDied","Data":"6c3d55783e3ee8336e7e0ad6946b84fd5e21905b9048ea92962933b2b953f584"} Oct 30 21:11:02.675122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.675097 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:11:02.763260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763227 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle\") pod \"ea35dc79-a590-4921-8efb-9e2084a5a401\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " Oct 30 21:11:02.763455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763286 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw\" (UniqueName: \"kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw\") pod \"ea35dc79-a590-4921-8efb-9e2084a5a401\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " Oct 30 21:11:02.763455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763316 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle\") pod \"ea35dc79-a590-4921-8efb-9e2084a5a401\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " Oct 30 21:11:02.763455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763345 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path\") pod \"ea35dc79-a590-4921-8efb-9e2084a5a401\" (UID: \"ea35dc79-a590-4921-8efb-9e2084a5a401\") " Oct 30 21:11:02.763735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763621 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle" pod "ea35dc79-a590-4921-8efb-9e2084a5a401" (UID: "ea35dc79-a590-4921-8efb-9e2084a5a401"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:02.763797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763727 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle" pod "ea35dc79-a590-4921-8efb-9e2084a5a401" (UID: "ea35dc79-a590-4921-8efb-9e2084a5a401"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:02.763797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.763743 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path" pod "ea35dc79-a590-4921-8efb-9e2084a5a401" (UID: "ea35dc79-a590-4921-8efb-9e2084a5a401"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:11:02.765095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.765073 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw" pod "ea35dc79-a590-4921-8efb-9e2084a5a401" (UID: "ea35dc79-a590-4921-8efb-9e2084a5a401"). InnerVolumeSpecName "kube-api-access-t8bzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:02.864689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.864611 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bzw\" (UniqueName: \"kubernetes.io/projected/ea35dc79-a590-4921-8efb-9e2084a5a401-kube-api-access-t8bzw\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:11:02.864689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.864642 5039 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-trusted-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:11:02.864689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.864674 5039 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ea35dc79-a590-4921-8efb-9e2084a5a401-archives-path\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:11:02.864689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:02.864683 5039 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea35dc79-a590-4921-8efb-9e2084a5a401-service-ca-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:11:03.544874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:03.544839 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" Oct 30 21:11:03.545046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:03.544847 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6b6wf-s5ppz" event={"ID":"ea35dc79-a590-4921-8efb-9e2084a5a401","Type":"ContainerDied","Data":"3dd78e1e62741e5192e19740e1195767807b985a891c10fee94983d05e0688d8"} Oct 30 21:11:03.545046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:03.544961 5039 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd78e1e62741e5192e19740e1195767807b985a891c10fee94983d05e0688d8" Oct 30 21:11:04.006750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:04.006718 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:04.145682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:04.145649 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:10.343320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:10.343284 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:25.183994 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:11:25.183955 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:11:26.628143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:26.628108 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:26.786683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:11:26.786640 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:55.183769 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:11:55.183733 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:12:10.459340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:12:10.459287 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:14.612193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:12:14.612151 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:25.184790 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:12:25.184747 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:12:55.183639 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:12:55.183594 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:13:01.220843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:13:01.220797 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:10.572058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:13:10.571959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:25.184902 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:13:25.184865 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:13:47.575112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:13:47.575067 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:54.517768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:13:54.517727 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.183984 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:13:55.183946 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:14:05.294629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:05.294588 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:05.435718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:05.435681 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:08.266792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:08.266752 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:08.425368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:08.425328 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:10.687310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:10.687274 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:17.311785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:17.311747 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.663010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:20.662952 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.811894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:20.811857 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.929134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:20.929052 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:14:20.930270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:20.930244 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:14:25.185171 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:14:25.185135 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:14:45.681982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:45.681860 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:47.944668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:47.944625 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:48.271418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:48.271316 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:52.828626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:52.828592 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:52.992781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:52.992743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.583084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.583047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.607803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.607775 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599"] Oct 30 21:14:53.608129 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608109 5039 cpu_manager.go:414] "containerMap: removing container" podUID="57293ead-fcc2-47e7-8033-d0beec008c51" containerName="registry" Oct 30 21:14:53.608174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608128 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57293ead-fcc2-47e7-8033-d0beec008c51" containerName="registry" Oct 30 21:14:53.608174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608154 5039 cpu_manager.go:414] "containerMap: removing container" podUID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" containerName="registry" Oct 30 21:14:53.608174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608159 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" containerName="registry" Oct 30 21:14:53.608174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608170 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d887faa8-91ab-4a8e-8b92-38a199ef7081" containerName="insights-gathering" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608175 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d887faa8-91ab-4a8e-8b92-38a199ef7081" containerName="insights-gathering" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608189 5039 cpu_manager.go:414] "containerMap: removing container" podUID="ea35dc79-a590-4921-8efb-9e2084a5a401" containerName="insights-gathering" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608194 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea35dc79-a590-4921-8efb-9e2084a5a401" containerName="insights-gathering" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608263 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea35dc79-a590-4921-8efb-9e2084a5a401" containerName="insights-gathering" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608272 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="57293ead-fcc2-47e7-8033-d0beec008c51" containerName="registry" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608280 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccf83b0-6149-4f2b-b3b1-db339a5a245e" containerName="registry" Oct 30 21:14:53.608374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.608286 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d887faa8-91ab-4a8e-8b92-38a199ef7081" containerName="insights-gathering" Oct 30 21:14:53.611552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.611534 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.632807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.632777 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-9554w\"" Oct 30 21:14:53.634365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.634340 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599"] Oct 30 21:14:53.642350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.642319 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:14:53.681446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681415 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c04da-8703-43db-8d77-dd7db44099bf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681458 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c04da-8703-43db-8d77-dd7db44099bf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681571 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c04da-8703-43db-8d77-dd7db44099bf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681605 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681627 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vxq\" (UniqueName: \"kubernetes.io/projected/687c04da-8703-43db-8d77-dd7db44099bf-kube-api-access-k8vxq\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681712 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681740 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681758 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.681852 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.681812 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.731368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.731337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.782549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782520 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782567 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782595 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782644 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782670 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c04da-8703-43db-8d77-dd7db44099bf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782699 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c04da-8703-43db-8d77-dd7db44099bf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782748 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c04da-8703-43db-8d77-dd7db44099bf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782782 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.782827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.782816 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vxq\" (UniqueName: \"kubernetes.io/projected/687c04da-8703-43db-8d77-dd7db44099bf-kube-api-access-k8vxq\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.783117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.783053 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.783194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.783175 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.783483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.783453 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.783563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.783451 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.783718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.783694 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687c04da-8703-43db-8d77-dd7db44099bf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.785167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.785135 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687c04da-8703-43db-8d77-dd7db44099bf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.785349 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.785330 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687c04da-8703-43db-8d77-dd7db44099bf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.798984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.798956 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687c04da-8703-43db-8d77-dd7db44099bf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.799153 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.799135 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vxq\" (UniqueName: \"kubernetes.io/projected/687c04da-8703-43db-8d77-dd7db44099bf-kube-api-access-k8vxq\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-rd599\" (UID: \"687c04da-8703-43db-8d77-dd7db44099bf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:53.870930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.870842 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.924729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:53.924704 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:54.228657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.228628 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599"] Oct 30 21:14:54.231169 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:14:54.231142 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687c04da_8703_43db_8d77_dd7db44099bf.slice/crio-0268b29eee7057a657cdff4ba44fb83a7024cf8ae202c60702f790ce158b2050 WatchSource:0}: Error finding container 0268b29eee7057a657cdff4ba44fb83a7024cf8ae202c60702f790ce158b2050: Status 404 returned error can't find the container with id 0268b29eee7057a657cdff4ba44fb83a7024cf8ae202c60702f790ce158b2050 Oct 30 21:14:54.233077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.233057 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:14:54.462132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.462099 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" event={"ID":"687c04da-8703-43db-8d77-dd7db44099bf","Type":"ContainerStarted","Data":"0268b29eee7057a657cdff4ba44fb83a7024cf8ae202c60702f790ce158b2050"} Oct 30 21:14:54.877938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.875854 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp"] Oct 30 21:14:54.880172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.880141 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.889644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.889620 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.889805 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.889667 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.889805 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.889696 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2547\" (UniqueName: \"kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.908871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.908841 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp"] Oct 30 21:14:54.910512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.910487 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:14:54.910653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.910563 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:14:54.910653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.910603 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5"] Oct 30 21:14:54.910754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.910674 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-mcxf8\"" Oct 30 21:14:54.914508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.914487 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:54.928941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.928916 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5"] Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990489 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990556 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990579 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpgh\" (UniqueName: \"kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990606 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990625 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2547\" (UniqueName: \"kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990661 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.990982 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.991476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.991159 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:54.998191 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:54.998170 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mcbfl\"" Oct 30 21:14:55.007984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.007956 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2547\" (UniqueName: \"kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:55.091910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.091868 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpgh\" (UniqueName: \"kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.092076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.092054 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.092142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.092119 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.092558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.092528 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.092738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.092596 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.108877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.108848 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpgh\" (UniqueName: \"kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.189572 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:14:55.189533 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:14:55.202232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.202207 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:14:55.237465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.237427 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:14:55.500232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.500074 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp"] Oct 30 21:14:55.502773 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:14:55.502740 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6017b3_cd42_4e12_a015_d0c265ea59cf.slice/crio-35f317b6fb051e877cf1aa0972af9edbc4602b19e652ff222e14a1ae6820f7ef WatchSource:0}: Error finding container 35f317b6fb051e877cf1aa0972af9edbc4602b19e652ff222e14a1ae6820f7ef: Status 404 returned error can't find the container with id 35f317b6fb051e877cf1aa0972af9edbc4602b19e652ff222e14a1ae6820f7ef Oct 30 21:14:55.538850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:55.538819 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5"] Oct 30 21:14:55.543500 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:14:55.543469 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf18dc46_a14d_466b_9395_638b3229a12a.slice/crio-772c67b0c418429171f85650fffa15596f8dd949c9256319bfc59012047ec90e WatchSource:0}: Error finding container 772c67b0c418429171f85650fffa15596f8dd949c9256319bfc59012047ec90e: Status 404 returned error can't find the container with id 772c67b0c418429171f85650fffa15596f8dd949c9256319bfc59012047ec90e Oct 30 21:14:56.475235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:56.475117 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" event={"ID":"bf18dc46-a14d-466b-9395-638b3229a12a","Type":"ContainerStarted","Data":"772c67b0c418429171f85650fffa15596f8dd949c9256319bfc59012047ec90e"} Oct 30 21:14:56.476896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:56.476853 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" event={"ID":"fa6017b3-cd42-4e12-a015-d0c265ea59cf","Type":"ContainerStarted","Data":"35f317b6fb051e877cf1aa0972af9edbc4602b19e652ff222e14a1ae6820f7ef"} Oct 30 21:14:56.917412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:56.916282 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:14:56.917412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:56.916413 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:14:56.917412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:56.916459 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:14:57.013504 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:57.013463 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:57.484318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:57.484213 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" event={"ID":"687c04da-8703-43db-8d77-dd7db44099bf","Type":"ContainerStarted","Data":"d66d3bde193f9bf111cd6e0e9d432babcf189bb8ba76b9668cdbd202fa4f6ae7"} Oct 30 21:14:57.541797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:57.541740 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" podStartSLOduration=1.859649942 podStartE2EDuration="4.541720662s" totalImagesPullingTime="2.68207072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:57.530900713 +0000 UTC m=+336.740840367" watchObservedRunningTime="2026-10-30 21:14:57.541720662 +0000 UTC m=+336.751660291" Oct 30 21:14:57.924791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:57.924748 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:58.931678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:58.931642 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:59.496863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:59.496835 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:14:59.498000 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:14:59.497977 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-rd599" Oct 30 21:15:02.511680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:02.511645 5039 generic.go:356] "Generic (PLEG): container finished" podID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerID="51c1cb6a0fc3d11db41279f58bee17d165571626dfa2714937c48dda06d0cc87" exitCode=0 Oct 30 21:15:02.512180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:02.511715 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" event={"ID":"fa6017b3-cd42-4e12-a015-d0c265ea59cf","Type":"ContainerDied","Data":"51c1cb6a0fc3d11db41279f58bee17d165571626dfa2714937c48dda06d0cc87"} Oct 30 21:15:02.513316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:02.513292 5039 generic.go:356] "Generic (PLEG): container finished" podID="bf18dc46-a14d-466b-9395-638b3229a12a" containerID="5efb178223b4fdc7b44ea20251740ca5f96855c797da74e69bdf3f615e2f1f59" exitCode=0 Oct 30 21:15:02.513434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:02.513369 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" event={"ID":"bf18dc46-a14d-466b-9395-638b3229a12a","Type":"ContainerDied","Data":"5efb178223b4fdc7b44ea20251740ca5f96855c797da74e69bdf3f615e2f1f59"} Oct 30 21:15:04.525156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:04.525124 5039 generic.go:356] "Generic (PLEG): container finished" podID="bf18dc46-a14d-466b-9395-638b3229a12a" containerID="ed7584ddff7f50beddb764e8539825a3431fcf63a34d8b3f051cd4864e600b74" exitCode=0 Oct 30 21:15:04.525623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:04.525204 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" event={"ID":"bf18dc46-a14d-466b-9395-638b3229a12a","Type":"ContainerDied","Data":"ed7584ddff7f50beddb764e8539825a3431fcf63a34d8b3f051cd4864e600b74"} Oct 30 21:15:04.527278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:04.527256 5039 generic.go:356] "Generic (PLEG): container finished" podID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerID="50b04df2e445ed6b64a8ff399dda13f60ec93565512e3bb49daecbe77e3fda98" exitCode=0 Oct 30 21:15:04.527341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:04.527320 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" event={"ID":"fa6017b3-cd42-4e12-a015-d0c265ea59cf","Type":"ContainerDied","Data":"50b04df2e445ed6b64a8ff399dda13f60ec93565512e3bb49daecbe77e3fda98"} Oct 30 21:15:10.810698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:10.810670 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:11.565321 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:11.565286 5039 generic.go:356] "Generic (PLEG): container finished" podID="bf18dc46-a14d-466b-9395-638b3229a12a" containerID="54ed6f2c7095d133ed28de6bde38070e075d5579acb34b1d2d344b3ff9406ec0" exitCode=0 Oct 30 21:15:11.565535 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:11.565334 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" event={"ID":"bf18dc46-a14d-466b-9395-638b3229a12a","Type":"ContainerDied","Data":"54ed6f2c7095d133ed28de6bde38070e075d5579acb34b1d2d344b3ff9406ec0"} Oct 30 21:15:11.567402 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:11.567365 5039 generic.go:356] "Generic (PLEG): container finished" podID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerID="cc3f52bec9779eaa9f9ef745db9b5988f872d57bc133f62a6e47a729035394c0" exitCode=0 Oct 30 21:15:11.567529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:11.567418 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" event={"ID":"fa6017b3-cd42-4e12-a015-d0c265ea59cf","Type":"ContainerDied","Data":"cc3f52bec9779eaa9f9ef745db9b5988f872d57bc133f62a6e47a729035394c0"} Oct 30 21:15:12.721519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.721488 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:15:12.724479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.724463 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:15:12.858064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.857988 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util\") pod \"bf18dc46-a14d-466b-9395-638b3229a12a\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " Oct 30 21:15:12.858064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858028 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle\") pod \"bf18dc46-a14d-466b-9395-638b3229a12a\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " Oct 30 21:15:12.858064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858048 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh\" (UniqueName: \"kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh\") pod \"bf18dc46-a14d-466b-9395-638b3229a12a\" (UID: \"bf18dc46-a14d-466b-9395-638b3229a12a\") " Oct 30 21:15:12.858300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858098 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util\") pod \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " Oct 30 21:15:12.858300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858121 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle\") pod \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " Oct 30 21:15:12.858300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858158 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547\" (UniqueName: \"kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547\") pod \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\" (UID: \"fa6017b3-cd42-4e12-a015-d0c265ea59cf\") " Oct 30 21:15:12.858655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858629 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle" pod "bf18dc46-a14d-466b-9395-638b3229a12a" (UID: "bf18dc46-a14d-466b-9395-638b3229a12a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:12.858754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.858730 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle" pod "fa6017b3-cd42-4e12-a015-d0c265ea59cf" (UID: "fa6017b3-cd42-4e12-a015-d0c265ea59cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:12.860441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.860412 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547" pod "fa6017b3-cd42-4e12-a015-d0c265ea59cf" (UID: "fa6017b3-cd42-4e12-a015-d0c265ea59cf"). InnerVolumeSpecName "kube-api-access-q2547". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:12.860625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.860604 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh" pod "bf18dc46-a14d-466b-9395-638b3229a12a" (UID: "bf18dc46-a14d-466b-9395-638b3229a12a"). InnerVolumeSpecName "kube-api-access-6qpgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:12.862508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.862486 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util" pod "bf18dc46-a14d-466b-9395-638b3229a12a" (UID: "bf18dc46-a14d-466b-9395-638b3229a12a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:12.863142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.863118 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util" pod "fa6017b3-cd42-4e12-a015-d0c265ea59cf" (UID: "fa6017b3-cd42-4e12-a015-d0c265ea59cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:12.959673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.959647 5039 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:15:12.959673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.959671 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2547\" (UniqueName: \"kubernetes.io/projected/fa6017b3-cd42-4e12-a015-d0c265ea59cf-kube-api-access-q2547\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:15:12.959813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.959682 5039 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-util\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:15:12.959813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.959692 5039 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf18dc46-a14d-466b-9395-638b3229a12a-bundle\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:15:12.959813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.959701 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qpgh\" (UniqueName: \"kubernetes.io/projected/bf18dc46-a14d-466b-9395-638b3229a12a-kube-api-access-6qpgh\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:15:12.959813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:12.959709 5039 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6017b3-cd42-4e12-a015-d0c265ea59cf-util\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:15:13.578290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:13.578260 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" Oct 30 21:15:13.578290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:13.578278 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xcff5" event={"ID":"bf18dc46-a14d-466b-9395-638b3229a12a","Type":"ContainerDied","Data":"772c67b0c418429171f85650fffa15596f8dd949c9256319bfc59012047ec90e"} Oct 30 21:15:13.578562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:13.578333 5039 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="772c67b0c418429171f85650fffa15596f8dd949c9256319bfc59012047ec90e" Oct 30 21:15:13.580179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:13.580153 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" event={"ID":"fa6017b3-cd42-4e12-a015-d0c265ea59cf","Type":"ContainerDied","Data":"35f317b6fb051e877cf1aa0972af9edbc4602b19e652ff222e14a1ae6820f7ef"} Oct 30 21:15:13.580293 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:13.580197 5039 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f317b6fb051e877cf1aa0972af9edbc4602b19e652ff222e14a1ae6820f7ef" Oct 30 21:15:13.580293 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:13.580200 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq58bp" Oct 30 21:15:25.184358 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:15:25.184318 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:15:32.560749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.560716 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j"] Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561052 5039 cpu_manager.go:414] "containerMap: removing container" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="util" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561064 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="util" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561074 5039 cpu_manager.go:414] "containerMap: removing container" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="pull" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561080 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="pull" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561090 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="util" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561095 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="util" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561103 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="extract" Oct 30 21:15:32.561117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561107 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="extract" Oct 30 21:15:32.561353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561124 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="pull" Oct 30 21:15:32.561353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561128 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="pull" Oct 30 21:15:32.561353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561136 5039 cpu_manager.go:414] "containerMap: removing container" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="extract" Oct 30 21:15:32.561353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561140 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="extract" Oct 30 21:15:32.561353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561196 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6017b3-cd42-4e12-a015-d0c265ea59cf" containerName="extract" Oct 30 21:15:32.561353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.561210 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf18dc46-a14d-466b-9395-638b3229a12a" containerName="extract" Oct 30 21:15:32.569721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.569694 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.605937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.605900 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j"] Oct 30 21:15:32.635170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.635121 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:32.661637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.661610 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:15:32.662308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.662289 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:15:32.662469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.662449 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:15:32.662764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.662748 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:15:32.662829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.662775 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:15:32.662829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.662785 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dggpd\"" Oct 30 21:15:32.727312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.727283 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.727479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.727319 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.727479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.727351 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjbm\" (UniqueName: \"kubernetes.io/projected/7119b8d8-fa63-478e-836d-93622546bda1-kube-api-access-ksjbm\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.727575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.727478 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.727575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.727505 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7119b8d8-fa63-478e-836d-93622546bda1-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.727575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.727521 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7119b8d8-fa63-478e-836d-93622546bda1-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.780106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.780076 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:32.828311 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828241 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.828311 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828280 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.828512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828325 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjbm\" (UniqueName: \"kubernetes.io/projected/7119b8d8-fa63-478e-836d-93622546bda1-kube-api-access-ksjbm\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.828512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828356 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.828512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828378 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7119b8d8-fa63-478e-836d-93622546bda1-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.828512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828426 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7119b8d8-fa63-478e-836d-93622546bda1-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.828908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.828878 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7119b8d8-fa63-478e-836d-93622546bda1-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.830630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.830602 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7119b8d8-fa63-478e-836d-93622546bda1-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.830726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.830661 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.830770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.830741 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.830815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.830798 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7119b8d8-fa63-478e-836d-93622546bda1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.844900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.844875 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjbm\" (UniqueName: \"kubernetes.io/projected/7119b8d8-fa63-478e-836d-93622546bda1-kube-api-access-ksjbm\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-mc48j\" (UID: \"7119b8d8-fa63-478e-836d-93622546bda1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.883987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.883966 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:32.935565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:32.935368 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:33.095308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:33.095210 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:33.215803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:33.215614 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j"] Oct 30 21:15:33.218966 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:15:33.218940 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7119b8d8_fa63_478e_836d_93622546bda1.slice/crio-06fdee3a02e48cff097dbc5647b932e087a5ca8c93bfbff48621dfd65118ca6d WatchSource:0}: Error finding container 06fdee3a02e48cff097dbc5647b932e087a5ca8c93bfbff48621dfd65118ca6d: Status 404 returned error can't find the container with id 06fdee3a02e48cff097dbc5647b932e087a5ca8c93bfbff48621dfd65118ca6d Oct 30 21:15:33.664678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:33.664642 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" event={"ID":"7119b8d8-fa63-478e-836d-93622546bda1","Type":"ContainerStarted","Data":"06fdee3a02e48cff097dbc5647b932e087a5ca8c93bfbff48621dfd65118ca6d"} Oct 30 21:15:35.375077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:35.375024 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:36.200824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:36.200743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:36.583626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:36.583593 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:36.679151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:36.679117 5039 generic.go:356] "Generic (PLEG): container finished" podID="7119b8d8-fa63-478e-836d-93622546bda1" containerID="e5fd3e335e72c693045187fd49c583f988f2e10c765fad3ddbb45000b1c8ed6c" exitCode=0 Oct 30 21:15:36.679320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:36.679196 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" event={"ID":"7119b8d8-fa63-478e-836d-93622546bda1","Type":"ContainerDied","Data":"e5fd3e335e72c693045187fd49c583f988f2e10c765fad3ddbb45000b1c8ed6c"} Oct 30 21:15:37.685789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:37.685757 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" event={"ID":"7119b8d8-fa63-478e-836d-93622546bda1","Type":"ContainerStarted","Data":"f28570d86e590e7a04a5269603419434af8a8406494dd6dbc171ef8d51029833"} Oct 30 21:15:37.686201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:37.685898 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:37.789279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:37.789230 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" podStartSLOduration=2.177070299 podStartE2EDuration="5.789216253s" totalImagesPullingTime="3.117295462s" totalInitContainerRuntime="494.850492ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:37.779312301 +0000 UTC m=+376.989251941" watchObservedRunningTime="2026-10-30 21:15:37.789216253 +0000 UTC m=+376.999155882" Oct 30 21:15:37.825246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:37.825214 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:38.264773 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:38.264727 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:39.631830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:39.631795 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:39.793322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:39.793286 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:43.945342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:43.945310 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr"] Oct 30 21:15:43.950170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:43.950148 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:43.972781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:43.972752 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:15:43.972908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:43.972807 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:15:43.972908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:43.972821 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rqzpl\"" Oct 30 21:15:43.975008 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:43.974988 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr"] Oct 30 21:15:44.015254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.015219 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6fw\" (UniqueName: \"kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-975qr\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.015371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.015349 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-975qr\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.116315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.116280 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-975qr\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.116465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.116330 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6fw\" (UniqueName: \"kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-975qr\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.116686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.116668 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-975qr\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.132992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.132965 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6fw\" (UniqueName: \"kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw\") pod \"kuadrant-operator-controller-manager-cf8469d7b-975qr\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.262347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.262263 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:44.573210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.573141 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr"] Oct 30 21:15:44.575450 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:15:44.575423 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1e8bfd_976f_47af_9146_8540e95b66cc.slice/crio-74e0d8da75e198b5584f760a1810741c693e23cd1c7ceb27d5e1bd563ab6a432 WatchSource:0}: Error finding container 74e0d8da75e198b5584f760a1810741c693e23cd1c7ceb27d5e1bd563ab6a432: Status 404 returned error can't find the container with id 74e0d8da75e198b5584f760a1810741c693e23cd1c7ceb27d5e1bd563ab6a432 Oct 30 21:15:44.714854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:44.714820 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" event={"ID":"8b1e8bfd-976f-47af-9146-8540e95b66cc","Type":"ContainerStarted","Data":"74e0d8da75e198b5584f760a1810741c693e23cd1c7ceb27d5e1bd563ab6a432"} Oct 30 21:15:45.391878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:45.391841 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:46.905823 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:46.905779 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:47.070452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:47.070414 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:48.691884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:48.691853 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-mc48j" Oct 30 21:15:49.182358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:49.182325 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:49.790155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:49.790117 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:51.680958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:51.680919 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:52.754053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.754014 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" event={"ID":"8b1e8bfd-976f-47af-9146-8540e95b66cc","Type":"ContainerStarted","Data":"28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022"} Oct 30 21:15:52.754450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.754173 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:15:52.811744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.811639 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" podStartSLOduration=1.8679658510000001 podStartE2EDuration="9.811625448s" totalImagesPullingTime="7.943659597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:52.801583161 +0000 UTC m=+392.011522803" watchObservedRunningTime="2026-10-30 21:15:52.811625448 +0000 UTC m=+392.021565077" Oct 30 21:15:52.882790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.882758 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7"] Oct 30 21:15:52.886885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.886864 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:52.903428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.903385 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:15:52.903428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.903385 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:15:52.903611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.903537 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6mjmn\"" Oct 30 21:15:52.908645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.908617 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7"] Oct 30 21:15:52.990291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.990256 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:52.995337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.995305 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbrt\" (UniqueName: \"kubernetes.io/projected/adcfe4a2-3563-44d4-a735-621e6b9b116d-kube-api-access-trbrt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7\" (UID: \"adcfe4a2-3563-44d4-a735-621e6b9b116d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:52.995563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:52.995539 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adcfe4a2-3563-44d4-a735-621e6b9b116d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7\" (UID: \"adcfe4a2-3563-44d4-a735-621e6b9b116d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:53.096383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.096301 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adcfe4a2-3563-44d4-a735-621e6b9b116d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7\" (UID: \"adcfe4a2-3563-44d4-a735-621e6b9b116d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:53.096573 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.096407 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbrt\" (UniqueName: \"kubernetes.io/projected/adcfe4a2-3563-44d4-a735-621e6b9b116d-kube-api-access-trbrt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7\" (UID: \"adcfe4a2-3563-44d4-a735-621e6b9b116d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:53.096757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.096735 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adcfe4a2-3563-44d4-a735-621e6b9b116d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7\" (UID: \"adcfe4a2-3563-44d4-a735-621e6b9b116d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:53.117926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.117884 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbrt\" (UniqueName: \"kubernetes.io/projected/adcfe4a2-3563-44d4-a735-621e6b9b116d-kube-api-access-trbrt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7\" (UID: \"adcfe4a2-3563-44d4-a735-621e6b9b116d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:53.201721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.201687 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" Oct 30 21:15:53.515884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.515851 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7"] Oct 30 21:15:53.528257 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:15:53.528226 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcfe4a2_3563_44d4_a735_621e6b9b116d.slice/crio-3bacedc2f237747b8f8d697050327c911f5cfa83d4381481db1364b880b5713e WatchSource:0}: Error finding container 3bacedc2f237747b8f8d697050327c911f5cfa83d4381481db1364b880b5713e: Status 404 returned error can't find the container with id 3bacedc2f237747b8f8d697050327c911f5cfa83d4381481db1364b880b5713e Oct 30 21:15:53.761001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.760963 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" event={"ID":"adcfe4a2-3563-44d4-a735-621e6b9b116d","Type":"ContainerStarted","Data":"3bacedc2f237747b8f8d697050327c911f5cfa83d4381481db1364b880b5713e"} Oct 30 21:15:53.775599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:53.775515 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:54.459828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:54.459788 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:55.186296 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:15:55.186252 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:15:55.681887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:55.681841 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:55.836243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:55.836129 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:56.777951 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:56.777910 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" event={"ID":"adcfe4a2-3563-44d4-a735-621e6b9b116d","Type":"ContainerStarted","Data":"d071793fab93521bdaa39fd772d5a816ab8b47e45dd0d38829b3da4fe2199beb"} Oct 30 21:15:56.813037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:56.813005 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:56.837907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:56.837844 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kxwd7" podStartSLOduration=2.222153325 podStartE2EDuration="4.837826717s" totalImagesPullingTime="2.615673392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:56.827032677 +0000 UTC m=+396.036972306" watchObservedRunningTime="2026-10-30 21:15:56.837826717 +0000 UTC m=+396.047766347" Oct 30 21:15:57.321235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.321202 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz"] Oct 30 21:15:57.324774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.324754 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:15:57.340059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.340034 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n6w7j\"" Oct 30 21:15:57.356381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.356345 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz"] Oct 30 21:15:57.434857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.434824 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78c78\" (UniqueName: \"kubernetes.io/projected/49251fad-bf0e-414b-8747-e2fb0f78f446-kube-api-access-78c78\") pod \"authorino-operator-7b7bb4bc77-xj7qz\" (UID: \"49251fad-bf0e-414b-8747-e2fb0f78f446\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:15:57.535866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.535836 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c78\" (UniqueName: \"kubernetes.io/projected/49251fad-bf0e-414b-8747-e2fb0f78f446-kube-api-access-78c78\") pod \"authorino-operator-7b7bb4bc77-xj7qz\" (UID: \"49251fad-bf0e-414b-8747-e2fb0f78f446\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:15:57.557286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.557255 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78c78\" (UniqueName: \"kubernetes.io/projected/49251fad-bf0e-414b-8747-e2fb0f78f446-kube-api-access-78c78\") pod \"authorino-operator-7b7bb4bc77-xj7qz\" (UID: \"49251fad-bf0e-414b-8747-e2fb0f78f446\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:15:57.636351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.636321 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:15:57.926665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:57.926550 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz"] Oct 30 21:15:57.929466 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:15:57.929432 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49251fad_bf0e_414b_8747_e2fb0f78f446.slice/crio-9505bd5b1c2bcc9ccc6ac5f02d3225c416c3bc2091c73d616df8964ec9f9b001 WatchSource:0}: Error finding container 9505bd5b1c2bcc9ccc6ac5f02d3225c416c3bc2091c73d616df8964ec9f9b001: Status 404 returned error can't find the container with id 9505bd5b1c2bcc9ccc6ac5f02d3225c416c3bc2091c73d616df8964ec9f9b001 Oct 30 21:15:58.048024 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:58.047981 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:58.210922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:58.210837 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:58.791636 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:58.791599 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" event={"ID":"49251fad-bf0e-414b-8747-e2fb0f78f446","Type":"ContainerStarted","Data":"9505bd5b1c2bcc9ccc6ac5f02d3225c416c3bc2091c73d616df8964ec9f9b001"} Oct 30 21:15:58.840462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:58.840412 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:58.995915 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:58.995868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:59.230177 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.230130 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:59.534907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.534812 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:59.631081 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.631043 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml"] Oct 30 21:15:59.634942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.634802 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.651553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.651525 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kk52r\"" Oct 30 21:15:59.663289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.663262 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml"] Oct 30 21:15:59.754844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.754809 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.754863 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.754987 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.755032 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.755121 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.755240 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jb9\" (UniqueName: \"kubernetes.io/projected/1ee76772-f17b-4258-85ea-8f9fc29bc20b-kube-api-access-s6jb9\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.755298 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.755332 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.755370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.755358 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.794547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.794469 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:59.798561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.798536 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" event={"ID":"49251fad-bf0e-414b-8747-e2fb0f78f446","Type":"ContainerStarted","Data":"464e97dce63045c8f39f5129b25a699479b738ef3e9ce01b8900707e8bb6d6e6"} Oct 30 21:15:59.798695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.798642 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:15:59.834626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.834583 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" podStartSLOduration=1.360384148 podStartE2EDuration="2.834571766s" totalImagesPullingTime="1.474187618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:59.834181724 +0000 UTC m=+399.044121353" watchObservedRunningTime="2026-10-30 21:15:59.834571766 +0000 UTC m=+399.044511394" Oct 30 21:15:59.855757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.855729 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.855910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.855793 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.855910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.855824 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.855910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.855850 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.855916 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jb9\" (UniqueName: \"kubernetes.io/projected/1ee76772-f17b-4258-85ea-8f9fc29bc20b-kube-api-access-s6jb9\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.855969 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856000 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856027 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856071 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856165 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856214 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856635 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856616 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856680 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.856721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.856691 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.858401 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.858366 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.858517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.858407 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.873663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.873613 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee76772-f17b-4258-85ea-8f9fc29bc20b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.873780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.873762 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jb9\" (UniqueName: \"kubernetes.io/projected/1ee76772-f17b-4258-85ea-8f9fc29bc20b-kube-api-access-s6jb9\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-ptnml\" (UID: \"1ee76772-f17b-4258-85ea-8f9fc29bc20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:15:59.943704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.943671 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:59.949537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:15:59.949516 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:16:00.088658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.088559 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:00.254624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.254600 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml"] Oct 30 21:16:00.257154 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:00.257126 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee76772_f17b_4258_85ea_8f9fc29bc20b.slice/crio-7c3a5c42a9e5309b2f980361ed9a64810c4e064d135489ee0109c9b34ec90eb6 WatchSource:0}: Error finding container 7c3a5c42a9e5309b2f980361ed9a64810c4e064d135489ee0109c9b34ec90eb6: Status 404 returned error can't find the container with id 7c3a5c42a9e5309b2f980361ed9a64810c4e064d135489ee0109c9b34ec90eb6 Oct 30 21:16:00.259306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.259274 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:00.259373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.259345 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:00.259432 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.259376 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:00.450079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.450047 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp"] Oct 30 21:16:00.454013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.453987 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.468936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.468905 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:16:00.468936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.468921 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:16:00.471482 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.471448 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp"] Oct 30 21:16:00.506110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.506069 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:00.507031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.507011 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6zrws\"" Oct 30 21:16:00.561660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.561631 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0423bc69-d1ce-4cbf-8598-30079f2fce0d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8mjkp\" (UID: \"0423bc69-d1ce-4cbf-8598-30079f2fce0d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.561818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.561669 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk7p\" (UniqueName: \"kubernetes.io/projected/0423bc69-d1ce-4cbf-8598-30079f2fce0d-kube-api-access-dsk7p\") pod \"cert-manager-cainjector-5c57bbd4fc-8mjkp\" (UID: \"0423bc69-d1ce-4cbf-8598-30079f2fce0d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.662706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.662671 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0423bc69-d1ce-4cbf-8598-30079f2fce0d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8mjkp\" (UID: \"0423bc69-d1ce-4cbf-8598-30079f2fce0d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.662860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.662711 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk7p\" (UniqueName: \"kubernetes.io/projected/0423bc69-d1ce-4cbf-8598-30079f2fce0d-kube-api-access-dsk7p\") pod \"cert-manager-cainjector-5c57bbd4fc-8mjkp\" (UID: \"0423bc69-d1ce-4cbf-8598-30079f2fce0d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.685027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.685001 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0423bc69-d1ce-4cbf-8598-30079f2fce0d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8mjkp\" (UID: \"0423bc69-d1ce-4cbf-8598-30079f2fce0d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.685197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.685101 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk7p\" (UniqueName: \"kubernetes.io/projected/0423bc69-d1ce-4cbf-8598-30079f2fce0d-kube-api-access-dsk7p\") pod \"cert-manager-cainjector-5c57bbd4fc-8mjkp\" (UID: \"0423bc69-d1ce-4cbf-8598-30079f2fce0d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.774312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.774247 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" Oct 30 21:16:00.806002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.805966 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" event={"ID":"1ee76772-f17b-4258-85ea-8f9fc29bc20b","Type":"ContainerStarted","Data":"07a2d529c9f9f5fd72bcfb3ea9826481a2eabdac69c624f4244a203ae29a4959"} Oct 30 21:16:00.806145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.806038 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" event={"ID":"1ee76772-f17b-4258-85ea-8f9fc29bc20b","Type":"ContainerStarted","Data":"7c3a5c42a9e5309b2f980361ed9a64810c4e064d135489ee0109c9b34ec90eb6"} Oct 30 21:16:00.841631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.841580 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" podStartSLOduration=1.8415658910000001 podStartE2EDuration="1.841565891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:00.840917848 +0000 UTC m=+400.050857476" watchObservedRunningTime="2026-10-30 21:16:00.841565891 +0000 UTC m=+400.051505520" Oct 30 21:16:00.950061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:00.950030 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:16:01.062881 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.062849 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp"] Oct 30 21:16:01.064777 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:01.064754 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0423bc69_d1ce_4cbf_8598_30079f2fce0d.slice/crio-054ddec98963549a29badd7fc4472f8e2d55e74836ae284173fe311a1e48af92 WatchSource:0}: Error finding container 054ddec98963549a29badd7fc4472f8e2d55e74836ae284173fe311a1e48af92: Status 404 returned error can't find the container with id 054ddec98963549a29badd7fc4472f8e2d55e74836ae284173fe311a1e48af92 Oct 30 21:16:01.180964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.180935 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:01.324433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.324308 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:01.770004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.769959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:01.812519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.812481 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" event={"ID":"0423bc69-d1ce-4cbf-8598-30079f2fce0d","Type":"ContainerStarted","Data":"054ddec98963549a29badd7fc4472f8e2d55e74836ae284173fe311a1e48af92"} Oct 30 21:16:01.940665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.940622 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:01.955988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:01.955957 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:16:02.191018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:02.190986 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.385219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:02.385170 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.542848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:02.542759 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.817240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:02.817165 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:16:02.818147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:02.818128 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-ptnml" Oct 30 21:16:03.763330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:03.763298 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:16:05.958461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:05.958428 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p"] Oct 30 21:16:05.962167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:05.962149 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:05.986430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:05.986405 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p"] Oct 30 21:16:05.994464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:05.994442 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jbp9b\"" Oct 30 21:16:06.011373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.011339 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7732615f-2c18-4474-8149-8a6798b4f6fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hcx4p\" (UID: \"7732615f-2c18-4474-8149-8a6798b4f6fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.011510 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.011402 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjdm\" (UniqueName: \"kubernetes.io/projected/7732615f-2c18-4474-8149-8a6798b4f6fe-kube-api-access-cfjdm\") pod \"cert-manager-webhook-5dbbbf75cc-hcx4p\" (UID: \"7732615f-2c18-4474-8149-8a6798b4f6fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.112420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.112370 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjdm\" (UniqueName: \"kubernetes.io/projected/7732615f-2c18-4474-8149-8a6798b4f6fe-kube-api-access-cfjdm\") pod \"cert-manager-webhook-5dbbbf75cc-hcx4p\" (UID: \"7732615f-2c18-4474-8149-8a6798b4f6fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.112591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.112514 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7732615f-2c18-4474-8149-8a6798b4f6fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hcx4p\" (UID: \"7732615f-2c18-4474-8149-8a6798b4f6fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.133957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.133925 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7732615f-2c18-4474-8149-8a6798b4f6fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hcx4p\" (UID: \"7732615f-2c18-4474-8149-8a6798b4f6fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.134099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.134042 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjdm\" (UniqueName: \"kubernetes.io/projected/7732615f-2c18-4474-8149-8a6798b4f6fe-kube-api-access-cfjdm\") pod \"cert-manager-webhook-5dbbbf75cc-hcx4p\" (UID: \"7732615f-2c18-4474-8149-8a6798b4f6fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.274541 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.274452 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:06.565441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.565383 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p"] Oct 30 21:16:06.568133 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:06.568103 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7732615f_2c18_4474_8149_8a6798b4f6fe.slice/crio-3b2610afb5d2fae7c3bfe0ce185c1a7f3f58e8149e32800f423fa124a96c0538 WatchSource:0}: Error finding container 3b2610afb5d2fae7c3bfe0ce185c1a7f3f58e8149e32800f423fa124a96c0538: Status 404 returned error can't find the container with id 3b2610afb5d2fae7c3bfe0ce185c1a7f3f58e8149e32800f423fa124a96c0538 Oct 30 21:16:06.836435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:06.836329 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" event={"ID":"7732615f-2c18-4474-8149-8a6798b4f6fe","Type":"ContainerStarted","Data":"3b2610afb5d2fae7c3bfe0ce185c1a7f3f58e8149e32800f423fa124a96c0538"} Oct 30 21:16:08.707349 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:08.707308 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:09.368854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:09.368815 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:09.506366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:09.506328 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:10.808515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.808486 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xj7qz" Oct 30 21:16:10.860738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.860703 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" event={"ID":"0423bc69-d1ce-4cbf-8598-30079f2fce0d","Type":"ContainerStarted","Data":"6c367f7c0c3f0c3dd91776a4b167e40d75af90e43d8cb8cff5dec825a6b5c039"} Oct 30 21:16:10.862558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.862532 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" event={"ID":"7732615f-2c18-4474-8149-8a6798b4f6fe","Type":"ContainerStarted","Data":"2e0019297469cfa170cd3d0e70f1235c77478d128c4f0a5af04f3773ae5c9719"} Oct 30 21:16:10.862696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.862660 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:10.895041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.894974 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8mjkp" podStartSLOduration=1.8895380099999999 podStartE2EDuration="10.894959981s" totalImagesPullingTime="9.005421971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:10.89365266 +0000 UTC m=+410.103592292" watchObservedRunningTime="2026-10-30 21:16:10.894959981 +0000 UTC m=+410.104899610" Oct 30 21:16:10.931227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.931192 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:10.933183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:10.933136 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" podStartSLOduration=2.439420627 podStartE2EDuration="5.933121277s" totalImagesPullingTime="3.49370065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:10.932426576 +0000 UTC m=+410.142366208" watchObservedRunningTime="2026-10-30 21:16:10.933121277 +0000 UTC m=+410.143060906" Oct 30 21:16:11.189918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:11.189881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:11.355224 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:11.355179 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:12.992748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:12.992702 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:13.148230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:13.148194 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:16.869960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:16.869932 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" Oct 30 21:16:17.855655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:17.855619 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:18.010193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.010159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:18.702406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.702363 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dx479"] Oct 30 21:16:18.707985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.707291 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.726143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.726113 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dx479"] Oct 30 21:16:18.730568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.730549 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gcbpl\"" Oct 30 21:16:18.827783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.827751 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjld5\" (UniqueName: \"kubernetes.io/projected/751ec479-4d55-4da6-bf4e-ea88c934deb2-kube-api-access-vjld5\") pod \"cert-manager-597b95cb84-dx479\" (UID: \"751ec479-4d55-4da6-bf4e-ea88c934deb2\") " pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.827937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.827790 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751ec479-4d55-4da6-bf4e-ea88c934deb2-bound-sa-token\") pod \"cert-manager-597b95cb84-dx479\" (UID: \"751ec479-4d55-4da6-bf4e-ea88c934deb2\") " pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.929314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.929275 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjld5\" (UniqueName: \"kubernetes.io/projected/751ec479-4d55-4da6-bf4e-ea88c934deb2-kube-api-access-vjld5\") pod \"cert-manager-597b95cb84-dx479\" (UID: \"751ec479-4d55-4da6-bf4e-ea88c934deb2\") " pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.929314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.929316 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751ec479-4d55-4da6-bf4e-ea88c934deb2-bound-sa-token\") pod \"cert-manager-597b95cb84-dx479\" (UID: \"751ec479-4d55-4da6-bf4e-ea88c934deb2\") " pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.953173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.953109 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751ec479-4d55-4da6-bf4e-ea88c934deb2-bound-sa-token\") pod \"cert-manager-597b95cb84-dx479\" (UID: \"751ec479-4d55-4da6-bf4e-ea88c934deb2\") " pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.953414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.953371 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjld5\" (UniqueName: \"kubernetes.io/projected/751ec479-4d55-4da6-bf4e-ea88c934deb2-kube-api-access-vjld5\") pod \"cert-manager-597b95cb84-dx479\" (UID: \"751ec479-4d55-4da6-bf4e-ea88c934deb2\") " pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:18.978016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.977986 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:18.981726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.981708 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:18.993730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.993711 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6hdk6\"" Oct 30 21:16:18.993886 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:18.993872 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:16:19.000766 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.000743 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:19.006780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.006728 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:19.020933 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.020912 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dx479" Oct 30 21:16:19.089458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.089432 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:19.130906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.130878 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrq9m\" (UniqueName: \"kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m\") pod \"limitador-limitador-74dfcfb894-9j7df\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.131034 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.131022 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file\") pod \"limitador-limitador-74dfcfb894-9j7df\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.231971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.231890 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file\") pod \"limitador-limitador-74dfcfb894-9j7df\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.231971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.231939 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrq9m\" (UniqueName: \"kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m\") pod \"limitador-limitador-74dfcfb894-9j7df\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.232619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.232596 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file\") pod \"limitador-limitador-74dfcfb894-9j7df\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.248497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.248472 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrq9m\" (UniqueName: \"kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m\") pod \"limitador-limitador-74dfcfb894-9j7df\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.295312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.295290 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:19.311786 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.311748 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dx479"] Oct 30 21:16:19.314814 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:19.314783 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod751ec479_4d55_4da6_bf4e_ea88c934deb2.slice/crio-2f31f5c75e89875b3a9f1e4e93d611009c58f7e1b195e4bad9f36be41193fefb WatchSource:0}: Error finding container 2f31f5c75e89875b3a9f1e4e93d611009c58f7e1b195e4bad9f36be41193fefb: Status 404 returned error can't find the container with id 2f31f5c75e89875b3a9f1e4e93d611009c58f7e1b195e4bad9f36be41193fefb Oct 30 21:16:19.590595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.590571 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:19.592452 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:19.592424 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490733ff_81e4_4534_9d3e_b423b79cbd47.slice/crio-771d716886787c5c20494150ef61cef75ac48c1d7291934547e5d17096307f73 WatchSource:0}: Error finding container 771d716886787c5c20494150ef61cef75ac48c1d7291934547e5d17096307f73: Status 404 returned error can't find the container with id 771d716886787c5c20494150ef61cef75ac48c1d7291934547e5d17096307f73 Oct 30 21:16:19.905717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.905678 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dx479" event={"ID":"751ec479-4d55-4da6-bf4e-ea88c934deb2","Type":"ContainerStarted","Data":"849661cd80baa2dadb0b070ae49d68bea30a11b5504bb18f8a02d3acac595773"} Oct 30 21:16:19.905717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.905719 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dx479" event={"ID":"751ec479-4d55-4da6-bf4e-ea88c934deb2","Type":"ContainerStarted","Data":"2f31f5c75e89875b3a9f1e4e93d611009c58f7e1b195e4bad9f36be41193fefb"} Oct 30 21:16:19.906890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.906867 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" event={"ID":"490733ff-81e4-4534-9d3e-b423b79cbd47","Type":"ContainerStarted","Data":"771d716886787c5c20494150ef61cef75ac48c1d7291934547e5d17096307f73"} Oct 30 21:16:19.939505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:19.939455 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dx479" podStartSLOduration=1.9394434299999999 podStartE2EDuration="1.93944343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:19.938900966 +0000 UTC m=+419.148840597" watchObservedRunningTime="2026-10-30 21:16:19.93944343 +0000 UTC m=+419.149383060" Oct 30 21:16:20.906207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:20.906171 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr"] Oct 30 21:16:20.906630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:20.906455 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" podUID="8b1e8bfd-976f-47af-9146-8540e95b66cc" containerName="manager" containerID="cri-o://28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022" gracePeriod=10 Oct 30 21:16:20.925309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:20.925281 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:20.925516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:20.925347 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" event={"ID":"490733ff-81e4-4534-9d3e-b423b79cbd47","Type":"ContainerStarted","Data":"396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519"} Oct 30 21:16:20.974380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:20.974335 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" podStartSLOduration=1.9595194839999999 podStartE2EDuration="2.974321573s" totalImagesPullingTime="1.014802089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:20.973507539 +0000 UTC m=+420.183447171" watchObservedRunningTime="2026-10-30 21:16:20.974321573 +0000 UTC m=+420.184261203" Oct 30 21:16:21.161581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.161519 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:16:21.251138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.251109 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw\" (UniqueName: \"kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw\") pod \"8b1e8bfd-976f-47af-9146-8540e95b66cc\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " Oct 30 21:16:21.251287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.251191 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume\") pod \"8b1e8bfd-976f-47af-9146-8540e95b66cc\" (UID: \"8b1e8bfd-976f-47af-9146-8540e95b66cc\") " Oct 30 21:16:21.251566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.251543 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume" pod "8b1e8bfd-976f-47af-9146-8540e95b66cc" (UID: "8b1e8bfd-976f-47af-9146-8540e95b66cc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:21.253340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.253315 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw" pod "8b1e8bfd-976f-47af-9146-8540e95b66cc" (UID: "8b1e8bfd-976f-47af-9146-8540e95b66cc"). InnerVolumeSpecName "kube-api-access-lv6fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:21.352032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.352000 5039 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1e8bfd-976f-47af-9146-8540e95b66cc-extensions-socket-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:16:21.352032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.352029 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6fw\" (UniqueName: \"kubernetes.io/projected/8b1e8bfd-976f-47af-9146-8540e95b66cc-kube-api-access-lv6fw\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:16:21.925013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.924981 5039 generic.go:356] "Generic (PLEG): container finished" podID="8b1e8bfd-976f-47af-9146-8540e95b66cc" containerID="28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022" exitCode=0 Oct 30 21:16:21.925494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.925020 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" Oct 30 21:16:21.925494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.925064 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" event={"ID":"8b1e8bfd-976f-47af-9146-8540e95b66cc","Type":"ContainerDied","Data":"28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022"} Oct 30 21:16:21.925494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.925113 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr" event={"ID":"8b1e8bfd-976f-47af-9146-8540e95b66cc","Type":"ContainerDied","Data":"74e0d8da75e198b5584f760a1810741c693e23cd1c7ceb27d5e1bd563ab6a432"} Oct 30 21:16:21.925494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.925144 5039 scope.go:122] "RemoveContainer" containerID="28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022" Oct 30 21:16:21.934720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.934705 5039 scope.go:122] "RemoveContainer" containerID="28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022" Oct 30 21:16:21.934992 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:16:21.934966 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022\": container with ID starting with 28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022 not found: ID does not exist" containerID="28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022" Oct 30 21:16:21.935033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.934997 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022"} err="failed to get container status \"28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022\": rpc error: code = NotFound desc = could not find container \"28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022\": container with ID starting with 28fd2b275ceb60286bc7c4da35ecb37d3799d072362702a47cbc7f3a5e310022 not found: ID does not exist" Oct 30 21:16:21.980256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.980229 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr"] Oct 30 21:16:21.989670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:21.989640 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-975qr"] Oct 30 21:16:22.660963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:22.660924 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:22.805527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:22.805492 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:22.919853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:22.919784 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1e8bfd-976f-47af-9146-8540e95b66cc" path="/var/lib/kubelet/pods/8b1e8bfd-976f-47af-9146-8540e95b66cc/volumes" Oct 30 21:16:23.186693 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:23.186601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:23.376079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:23.376046 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:25.184925 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:16:25.184888 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:16:29.114911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:29.114877 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:29.264574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:29.264527 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:31.926381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:31.926353 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:33.885892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:33.885853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:34.034973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:34.034929 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:37.970755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.970718 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:16:37.971166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.971150 5039 cpu_manager.go:414] "containerMap: removing container" podUID="8b1e8bfd-976f-47af-9146-8540e95b66cc" containerName="manager" Oct 30 21:16:37.971220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.971169 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1e8bfd-976f-47af-9146-8540e95b66cc" containerName="manager" Oct 30 21:16:37.971257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.971238 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1e8bfd-976f-47af-9146-8540e95b66cc" containerName="manager" Oct 30 21:16:37.975791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.975775 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:37.978849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.978825 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d31c935f-6b9a-4ace-a486-16dcbb32e567-config-file\") pod \"limitador-limitador-7c9b87d574-9ngr9\" (UID: \"d31c935f-6b9a-4ace-a486-16dcbb32e567\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:37.978950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.978867 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b979x\" (UniqueName: \"kubernetes.io/projected/d31c935f-6b9a-4ace-a486-16dcbb32e567-kube-api-access-b979x\") pod \"limitador-limitador-7c9b87d574-9ngr9\" (UID: \"d31c935f-6b9a-4ace-a486-16dcbb32e567\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:37.992628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:37.992601 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:16:38.051080 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.051051 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:16:38.079713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.079682 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b979x\" (UniqueName: \"kubernetes.io/projected/d31c935f-6b9a-4ace-a486-16dcbb32e567-kube-api-access-b979x\") pod \"limitador-limitador-7c9b87d574-9ngr9\" (UID: \"d31c935f-6b9a-4ace-a486-16dcbb32e567\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:38.080082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.079810 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d31c935f-6b9a-4ace-a486-16dcbb32e567-config-file\") pod \"limitador-limitador-7c9b87d574-9ngr9\" (UID: \"d31c935f-6b9a-4ace-a486-16dcbb32e567\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:38.080500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.080478 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d31c935f-6b9a-4ace-a486-16dcbb32e567-config-file\") pod \"limitador-limitador-7c9b87d574-9ngr9\" (UID: \"d31c935f-6b9a-4ace-a486-16dcbb32e567\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:38.096427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.096377 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b979x\" (UniqueName: \"kubernetes.io/projected/d31c935f-6b9a-4ace-a486-16dcbb32e567-kube-api-access-b979x\") pod \"limitador-limitador-7c9b87d574-9ngr9\" (UID: \"d31c935f-6b9a-4ace-a486-16dcbb32e567\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:38.179697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.179663 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:38.289942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.289854 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:38.330462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.330429 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:38.579256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.579229 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:16:38.583941 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:38.583915 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31c935f_6b9a_4ace_a486_16dcbb32e567.slice/crio-d283eefccd8d56ab178ecb7a750b7b36998b53a056858cac90a091fa94a11786 WatchSource:0}: Error finding container d283eefccd8d56ab178ecb7a750b7b36998b53a056858cac90a091fa94a11786: Status 404 returned error can't find the container with id d283eefccd8d56ab178ecb7a750b7b36998b53a056858cac90a091fa94a11786 Oct 30 21:16:38.678802 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.678771 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:38.816491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:38.816457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:39.023495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:39.023035 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" event={"ID":"d31c935f-6b9a-4ace-a486-16dcbb32e567","Type":"ContainerStarted","Data":"673a636f1e43995b7d7d90da8fdf9a6779a05ec267ac21e2e23f79d14a63a59a"} Oct 30 21:16:39.023495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:39.023171 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" event={"ID":"d31c935f-6b9a-4ace-a486-16dcbb32e567","Type":"ContainerStarted","Data":"d283eefccd8d56ab178ecb7a750b7b36998b53a056858cac90a091fa94a11786"} Oct 30 21:16:39.023495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:39.023434 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:39.029548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:39.029518 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:39.057580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:39.057536 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" podStartSLOduration=2.057525636 podStartE2EDuration="2.057525636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:39.056708879 +0000 UTC m=+438.266648508" watchObservedRunningTime="2026-10-30 21:16:39.057525636 +0000 UTC m=+438.267465265" Oct 30 21:16:39.190872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:39.190843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:43.681876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:43.681837 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:43.835013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:43.834979 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:44.088828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:44.088742 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:44.231670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:44.231635 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:44.500133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:44.500098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:44.642983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:44.642943 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:50.029309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.029278 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9ngr9" Oct 30 21:16:50.089708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.089677 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:50.089926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.089902 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" podUID="490733ff-81e4-4534-9d3e-b423b79cbd47" containerName="limitador" containerID="cri-o://396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519" gracePeriod=30 Oct 30 21:16:50.534263 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.533566 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:50.650362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.650299 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:50.666160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.666133 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file\" (UniqueName: \"kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file\") pod \"490733ff-81e4-4534-9d3e-b423b79cbd47\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " Oct 30 21:16:50.666295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.666221 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m\" (UniqueName: \"kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m\") pod \"490733ff-81e4-4534-9d3e-b423b79cbd47\" (UID: \"490733ff-81e4-4534-9d3e-b423b79cbd47\") " Oct 30 21:16:50.666520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.666487 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file" pod "490733ff-81e4-4534-9d3e-b423b79cbd47" (UID: "490733ff-81e4-4534-9d3e-b423b79cbd47"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:50.668176 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.668145 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:50.668607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.668586 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m" pod "490733ff-81e4-4534-9d3e-b423b79cbd47" (UID: "490733ff-81e4-4534-9d3e-b423b79cbd47"). InnerVolumeSpecName "kube-api-access-lrq9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:50.767179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.767143 5039 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/490733ff-81e4-4534-9d3e-b423b79cbd47-config-file\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:16:50.767179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:50.767171 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrq9m\" (UniqueName: \"kubernetes.io/projected/490733ff-81e4-4534-9d3e-b423b79cbd47-kube-api-access-lrq9m\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:16:51.079664 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.079621 5039 generic.go:356] "Generic (PLEG): container finished" podID="490733ff-81e4-4534-9d3e-b423b79cbd47" containerID="396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519" exitCode=0 Oct 30 21:16:51.080094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.079678 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" Oct 30 21:16:51.080094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.079707 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" event={"ID":"490733ff-81e4-4534-9d3e-b423b79cbd47","Type":"ContainerDied","Data":"396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519"} Oct 30 21:16:51.080094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.079781 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9j7df" event={"ID":"490733ff-81e4-4534-9d3e-b423b79cbd47","Type":"ContainerDied","Data":"771d716886787c5c20494150ef61cef75ac48c1d7291934547e5d17096307f73"} Oct 30 21:16:51.080094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.079825 5039 scope.go:122] "RemoveContainer" containerID="396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519" Oct 30 21:16:51.088291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.088264 5039 scope.go:122] "RemoveContainer" containerID="396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519" Oct 30 21:16:51.088594 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:16:51.088574 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519\": container with ID starting with 396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519 not found: ID does not exist" containerID="396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519" Oct 30 21:16:51.088641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.088604 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519"} err="failed to get container status \"396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519\": rpc error: code = NotFound desc = could not find container \"396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519\": container with ID starting with 396116f8a31a25e4f40b42ca9c40806319180d4d86da6d1bbf855c8b3d44b519 not found: ID does not exist" Oct 30 21:16:51.121783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.121752 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:51.131827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:51.131798 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9j7df"] Oct 30 21:16:52.506722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:52.506686 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:52.655791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:52.655742 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:52.919764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:52.919732 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490733ff-81e4-4534-9d3e-b423b79cbd47" path="/var/lib/kubelet/pods/490733ff-81e4-4534-9d3e-b423b79cbd47/volumes" Oct 30 21:16:53.181803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.181730 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p"] Oct 30 21:16:53.182294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.182278 5039 cpu_manager.go:414] "containerMap: removing container" podUID="490733ff-81e4-4534-9d3e-b423b79cbd47" containerName="limitador" Oct 30 21:16:53.182333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.182297 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490733ff-81e4-4534-9d3e-b423b79cbd47" containerName="limitador" Oct 30 21:16:53.182474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.182462 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="490733ff-81e4-4534-9d3e-b423b79cbd47" containerName="limitador" Oct 30 21:16:53.186721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.186701 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.202782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.202755 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 21:16:53.212137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.212111 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:16:53.212262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.212190 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-45x8w\"" Oct 30 21:16:53.212262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.212225 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:16:53.214933 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.214910 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p"] Oct 30 21:16:53.285936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.285905 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/96dc590d-ce5f-473e-b427-53438f710fdb-data\") pod \"postgres-7cdbf6dc44-td45p\" (UID: \"96dc590d-ce5f-473e-b427-53438f710fdb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.286076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.285965 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flw2z\" (UniqueName: \"kubernetes.io/projected/96dc590d-ce5f-473e-b427-53438f710fdb-kube-api-access-flw2z\") pod \"postgres-7cdbf6dc44-td45p\" (UID: \"96dc590d-ce5f-473e-b427-53438f710fdb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.333256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.333213 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:53.386594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.386567 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/96dc590d-ce5f-473e-b427-53438f710fdb-data\") pod \"postgres-7cdbf6dc44-td45p\" (UID: \"96dc590d-ce5f-473e-b427-53438f710fdb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.386723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.386619 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flw2z\" (UniqueName: \"kubernetes.io/projected/96dc590d-ce5f-473e-b427-53438f710fdb-kube-api-access-flw2z\") pod \"postgres-7cdbf6dc44-td45p\" (UID: \"96dc590d-ce5f-473e-b427-53438f710fdb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.386939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.386921 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/96dc590d-ce5f-473e-b427-53438f710fdb-data\") pod \"postgres-7cdbf6dc44-td45p\" (UID: \"96dc590d-ce5f-473e-b427-53438f710fdb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.402752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.402727 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flw2z\" (UniqueName: \"kubernetes.io/projected/96dc590d-ce5f-473e-b427-53438f710fdb-kube-api-access-flw2z\") pod \"postgres-7cdbf6dc44-td45p\" (UID: \"96dc590d-ce5f-473e-b427-53438f710fdb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.506021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.505953 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:53.793689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:53.793665 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p"] Oct 30 21:16:53.796190 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:16:53.796165 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dc590d_ce5f_473e_b427_53438f710fdb.slice/crio-3ae866d79abc944f8c8e7b5818886c6657847101389362b59dc0d97cfa4992e9 WatchSource:0}: Error finding container 3ae866d79abc944f8c8e7b5818886c6657847101389362b59dc0d97cfa4992e9: Status 404 returned error can't find the container with id 3ae866d79abc944f8c8e7b5818886c6657847101389362b59dc0d97cfa4992e9 Oct 30 21:16:54.095671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:54.095593 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" event={"ID":"96dc590d-ce5f-473e-b427-53438f710fdb","Type":"ContainerStarted","Data":"3ae866d79abc944f8c8e7b5818886c6657847101389362b59dc0d97cfa4992e9"} Oct 30 21:16:55.193348 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:16:55.193312 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:16:55.302878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:55.302846 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:55.472886 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:55.472789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:59.124839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:59.124800 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" event={"ID":"96dc590d-ce5f-473e-b427-53438f710fdb","Type":"ContainerStarted","Data":"ad65ea4a02354348ad8247afd3950051aa049379d02d430c68cbf9ebb39b93d6"} Oct 30 21:16:59.125268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:59.124980 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:16:59.159737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:16:59.159691 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" podStartSLOduration=1.385523627 podStartE2EDuration="6.159679219s" totalImagesPullingTime="4.774155592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:59.158443558 +0000 UTC m=+458.368383188" watchObservedRunningTime="2026-10-30 21:16:59.159679219 +0000 UTC m=+458.369618848" Oct 30 21:17:00.099405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:00.099347 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:00.331588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:00.331524 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:03.509155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.509119 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:03.668924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.668884 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:03.772242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.772159 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l"] Oct 30 21:17:03.775667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.775645 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:03.792172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.792153 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:17:03.804578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.804550 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l"] Oct 30 21:17:03.840569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.840545 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j6hwv\"" Oct 30 21:17:03.848904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.848879 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:17:03.849037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.848885 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:17:03.856644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.856620 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:03.877573 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.877546 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hfx9l\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:03.877683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.877607 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnxd\" (UniqueName: \"kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd\") pod \"llmisvc-controller-manager-869d5dcc5d-hfx9l\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:03.979112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.979077 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnxd\" (UniqueName: \"kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd\") pod \"llmisvc-controller-manager-869d5dcc5d-hfx9l\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:03.979267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.979138 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hfx9l\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:03.981565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.981538 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hfx9l\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:03.996504 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:03.996484 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnxd\" (UniqueName: \"kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd\") pod \"llmisvc-controller-manager-869d5dcc5d-hfx9l\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:04.010031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:04.010003 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:04.088110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:04.088020 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:04.230797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:04.230755 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:04.375907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:04.375877 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l"] Oct 30 21:17:04.378078 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:17:04.378049 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod007ac525_f729_4832_ac3c_4d6270af60c6.slice/crio-a67eeecc3860c89f4498de5fa8a792fd0fcd8a4fbc96ed7e84a90b2a41d4aa10 WatchSource:0}: Error finding container a67eeecc3860c89f4498de5fa8a792fd0fcd8a4fbc96ed7e84a90b2a41d4aa10: Status 404 returned error can't find the container with id a67eeecc3860c89f4498de5fa8a792fd0fcd8a4fbc96ed7e84a90b2a41d4aa10 Oct 30 21:17:05.162058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:05.162013 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" event={"ID":"007ac525-f729-4832-ac3c-4d6270af60c6","Type":"ContainerStarted","Data":"a67eeecc3860c89f4498de5fa8a792fd0fcd8a4fbc96ed7e84a90b2a41d4aa10"} Oct 30 21:17:05.167830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:05.167802 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-td45p" Oct 30 21:17:05.936271 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:05.936234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:06.071234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:06.071116 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:06.208330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:06.208246 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:06.354077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:06.354039 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:07.424633 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:07.424603 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:07.621016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:07.620936 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:08.178202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:08.178164 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" event={"ID":"007ac525-f729-4832-ac3c-4d6270af60c6","Type":"ContainerStarted","Data":"9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be"} Oct 30 21:17:08.178416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:08.178269 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:08.210211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:08.210162 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" podStartSLOduration=2.339526268 podStartE2EDuration="5.2101501s" totalImagesPullingTime="2.870623832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:08.209704156 +0000 UTC m=+467.419643786" watchObservedRunningTime="2026-10-30 21:17:08.2101501 +0000 UTC m=+467.420089729" Oct 30 21:17:11.036350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:11.036310 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:12.412813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:12.412774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:12.557023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:12.556973 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:12.695092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:12.694988 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:12.906620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:12.906566 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.111658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:16.111622 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.283738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:16.283705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:17.996972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:17.996934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:18.155524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:18.155486 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:19.053206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:19.053152 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:19.213851 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:19.213818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:25.183860 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:17:25.183802 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:17:39.185658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:39.185568 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:17:48.243588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:48.243550 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:48.407447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:48.407386 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:49.732310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.732274 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:49.882171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.882136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:49.931916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.931889 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk"] Oct 30 21:17:49.938975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.938948 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:49.961622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.961590 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:17:49.961622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.961593 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:17:49.961809 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.961651 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q645h\"" Oct 30 21:17:49.963454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.963430 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk"] Oct 30 21:17:49.981122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.981099 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:49.981232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.981154 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrsm\" (UniqueName: \"kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:49.981232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.981178 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:49.981232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:49.981218 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.021192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.021093 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:50.082381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.082344 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.082580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.082408 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrsm\" (UniqueName: \"kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.082580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.082448 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.082580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.082464 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.083118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.083095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.085032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.085006 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.085126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.085006 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.098645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.098622 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrsm\" (UniqueName: \"kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm\") pod \"maas-api-5dd55849b-rxtlk\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.251777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.251742 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:50.541771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:50.541717 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk"] Oct 30 21:17:50.544047 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:17:50.544015 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f25dba_b014_41c6_ad03_ceb9eba2f777.slice/crio-de24f622850c7f04542c797bc6fe2429d78e14a4011ad8197bb39f6af3d84070 WatchSource:0}: Error finding container de24f622850c7f04542c797bc6fe2429d78e14a4011ad8197bb39f6af3d84070: Status 404 returned error can't find the container with id de24f622850c7f04542c797bc6fe2429d78e14a4011ad8197bb39f6af3d84070 Oct 30 21:17:51.378036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:51.377991 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" event={"ID":"e0f25dba-b014-41c6-ad03-ceb9eba2f777","Type":"ContainerStarted","Data":"de24f622850c7f04542c797bc6fe2429d78e14a4011ad8197bb39f6af3d84070"} Oct 30 21:17:51.542919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:51.542866 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:17:53.390719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:53.390681 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" event={"ID":"e0f25dba-b014-41c6-ad03-ceb9eba2f777","Type":"ContainerStarted","Data":"5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8"} Oct 30 21:17:53.391278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:53.390819 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:53.425423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:53.425354 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" podStartSLOduration=2.192486847 podStartE2EDuration="4.425340455s" totalImagesPullingTime="2.232853608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:53.423939835 +0000 UTC m=+512.633879464" watchObservedRunningTime="2026-10-30 21:17:53.425340455 +0000 UTC m=+512.635280084" Oct 30 21:17:55.183701 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:17:55.183660 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:17:59.400595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:59.400562 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:17:59.947374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:17:59.947335 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:00.085026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:00.084986 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:04.117930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.117876 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls"] Oct 30 21:18:04.121939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.121913 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.140414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.140365 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls"] Oct 30 21:18:04.208223 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.208190 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2sm\" (UniqueName: \"kubernetes.io/projected/c112dc6f-ef7f-4aaf-9e71-38183285f0d8-kube-api-access-sf2sm\") pod \"llmisvc-controller-manager-6466c554d8-pg4ls\" (UID: \"c112dc6f-ef7f-4aaf-9e71-38183285f0d8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.208356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.208242 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c112dc6f-ef7f-4aaf-9e71-38183285f0d8-cert\") pod \"llmisvc-controller-manager-6466c554d8-pg4ls\" (UID: \"c112dc6f-ef7f-4aaf-9e71-38183285f0d8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.309577 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.309540 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2sm\" (UniqueName: \"kubernetes.io/projected/c112dc6f-ef7f-4aaf-9e71-38183285f0d8-kube-api-access-sf2sm\") pod \"llmisvc-controller-manager-6466c554d8-pg4ls\" (UID: \"c112dc6f-ef7f-4aaf-9e71-38183285f0d8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.309756 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.309599 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c112dc6f-ef7f-4aaf-9e71-38183285f0d8-cert\") pod \"llmisvc-controller-manager-6466c554d8-pg4ls\" (UID: \"c112dc6f-ef7f-4aaf-9e71-38183285f0d8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.312055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.312028 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c112dc6f-ef7f-4aaf-9e71-38183285f0d8-cert\") pod \"llmisvc-controller-manager-6466c554d8-pg4ls\" (UID: \"c112dc6f-ef7f-4aaf-9e71-38183285f0d8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.328386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.328365 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2sm\" (UniqueName: \"kubernetes.io/projected/c112dc6f-ef7f-4aaf-9e71-38183285f0d8-kube-api-access-sf2sm\") pod \"llmisvc-controller-manager-6466c554d8-pg4ls\" (UID: \"c112dc6f-ef7f-4aaf-9e71-38183285f0d8\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.435231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.435200 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:04.724367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:04.724336 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls"] Oct 30 21:18:04.726198 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:18:04.726165 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc112dc6f_ef7f_4aaf_9e71_38183285f0d8.slice/crio-505f76edb93eb79462dcdf7636f6f9bb681b7abae0d8c4dcd923c05454688498 WatchSource:0}: Error finding container 505f76edb93eb79462dcdf7636f6f9bb681b7abae0d8c4dcd923c05454688498: Status 404 returned error can't find the container with id 505f76edb93eb79462dcdf7636f6f9bb681b7abae0d8c4dcd923c05454688498 Oct 30 21:18:05.448612 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:05.448576 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" event={"ID":"c112dc6f-ef7f-4aaf-9e71-38183285f0d8","Type":"ContainerStarted","Data":"daaeb6397c3848d1cec629cdb243007604d0c7836154fa2bc9657f3c1c48e1c9"} Oct 30 21:18:05.448993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:05.448628 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" event={"ID":"c112dc6f-ef7f-4aaf-9e71-38183285f0d8","Type":"ContainerStarted","Data":"505f76edb93eb79462dcdf7636f6f9bb681b7abae0d8c4dcd923c05454688498"} Oct 30 21:18:05.448993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:05.448674 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:05.482130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:05.482077 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" podStartSLOduration=1.1428655189999999 podStartE2EDuration="1.482059132s" totalImagesPullingTime="339.193613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:05.480813223 +0000 UTC m=+524.690752853" watchObservedRunningTime="2026-10-30 21:18:05.482059132 +0000 UTC m=+524.691998763" Oct 30 21:18:11.156100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.156058 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:11.478285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.478199 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk"] Oct 30 21:18:11.478498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.478467 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" podUID="e0f25dba-b014-41c6-ad03-ceb9eba2f777" containerName="maas-api" containerID="cri-o://5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8" gracePeriod=30 Oct 30 21:18:11.725698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.725668 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:18:11.773848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.773776 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca\") pod \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " Oct 30 21:18:11.773848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.773847 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm\" (UniqueName: \"kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm\") pod \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " Oct 30 21:18:11.774076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.773893 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls\") pod \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " Oct 30 21:18:11.774076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.773916 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs\") pod \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\" (UID: \"e0f25dba-b014-41c6-ad03-ceb9eba2f777\") " Oct 30 21:18:11.774224 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.774194 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca" pod "e0f25dba-b014-41c6-ad03-ceb9eba2f777" (UID: "e0f25dba-b014-41c6-ad03-ceb9eba2f777"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:11.776103 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.776073 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls" pod "e0f25dba-b014-41c6-ad03-ceb9eba2f777" (UID: "e0f25dba-b014-41c6-ad03-ceb9eba2f777"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:11.776212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.776116 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs" pod "e0f25dba-b014-41c6-ad03-ceb9eba2f777" (UID: "e0f25dba-b014-41c6-ad03-ceb9eba2f777"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:11.776212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.776146 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm" pod "e0f25dba-b014-41c6-ad03-ceb9eba2f777" (UID: "e0f25dba-b014-41c6-ad03-ceb9eba2f777"). InnerVolumeSpecName "kube-api-access-kzrsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:11.875472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.875437 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f25dba-b014-41c6-ad03-ceb9eba2f777-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:18:11.875472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.875465 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrsm\" (UniqueName: \"kubernetes.io/projected/e0f25dba-b014-41c6-ad03-ceb9eba2f777-kube-api-access-kzrsm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:18:11.875472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.875476 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:18:11.875684 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:11.875485 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f25dba-b014-41c6-ad03-ceb9eba2f777-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:18:12.483917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.483878 5039 generic.go:356] "Generic (PLEG): container finished" podID="e0f25dba-b014-41c6-ad03-ceb9eba2f777" containerID="5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8" exitCode=0 Oct 30 21:18:12.484346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.483922 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" Oct 30 21:18:12.484346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.483966 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" event={"ID":"e0f25dba-b014-41c6-ad03-ceb9eba2f777","Type":"ContainerDied","Data":"5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8"} Oct 30 21:18:12.484346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.484022 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk" event={"ID":"e0f25dba-b014-41c6-ad03-ceb9eba2f777","Type":"ContainerDied","Data":"de24f622850c7f04542c797bc6fe2429d78e14a4011ad8197bb39f6af3d84070"} Oct 30 21:18:12.484346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.484046 5039 scope.go:122] "RemoveContainer" containerID="5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8" Oct 30 21:18:12.493198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.493181 5039 scope.go:122] "RemoveContainer" containerID="5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8" Oct 30 21:18:12.493455 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:18:12.493437 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8\": container with ID starting with 5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8 not found: ID does not exist" containerID="5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8" Oct 30 21:18:12.493507 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.493463 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8"} err="failed to get container status \"5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8\": rpc error: code = NotFound desc = could not find container \"5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8\": container with ID starting with 5126cf56b3d023d83e59d043d8042e2406df6e04f5db1912855ce0a6aa1e96e8 not found: ID does not exist" Oct 30 21:18:12.527440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.527418 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk"] Oct 30 21:18:12.536883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.536860 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd55849b-rxtlk"] Oct 30 21:18:12.920402 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:12.920365 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f25dba-b014-41c6-ad03-ceb9eba2f777" path="/var/lib/kubelet/pods/e0f25dba-b014-41c6-ad03-ceb9eba2f777/volumes" Oct 30 21:18:25.183761 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:18:25.183720 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:18:36.456934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.456902 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-pg4ls" Oct 30 21:18:36.517562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.517523 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l"] Oct 30 21:18:36.517749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.517718 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" podUID="007ac525-f729-4832-ac3c-4d6270af60c6" containerName="manager" containerID="cri-o://9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be" gracePeriod=30 Oct 30 21:18:36.769754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.769731 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:18:36.897022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.896993 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert\" (UniqueName: \"kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert\") pod \"007ac525-f729-4832-ac3c-4d6270af60c6\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " Oct 30 21:18:36.897022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.897026 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd\" (UniqueName: \"kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd\") pod \"007ac525-f729-4832-ac3c-4d6270af60c6\" (UID: \"007ac525-f729-4832-ac3c-4d6270af60c6\") " Oct 30 21:18:36.899168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.899138 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd" pod "007ac525-f729-4832-ac3c-4d6270af60c6" (UID: "007ac525-f729-4832-ac3c-4d6270af60c6"). InnerVolumeSpecName "kube-api-access-2wnxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:36.899244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.899221 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert" pod "007ac525-f729-4832-ac3c-4d6270af60c6" (UID: "007ac525-f729-4832-ac3c-4d6270af60c6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:36.998558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.998510 5039 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/007ac525-f729-4832-ac3c-4d6270af60c6-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:18:36.998558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:36.998553 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wnxd\" (UniqueName: \"kubernetes.io/projected/007ac525-f729-4832-ac3c-4d6270af60c6-kube-api-access-2wnxd\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:18:37.600574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.600539 5039 generic.go:356] "Generic (PLEG): container finished" podID="007ac525-f729-4832-ac3c-4d6270af60c6" containerID="9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be" exitCode=0 Oct 30 21:18:37.600994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.600581 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" event={"ID":"007ac525-f729-4832-ac3c-4d6270af60c6","Type":"ContainerDied","Data":"9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be"} Oct 30 21:18:37.600994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.600603 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" Oct 30 21:18:37.600994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.600625 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l" event={"ID":"007ac525-f729-4832-ac3c-4d6270af60c6","Type":"ContainerDied","Data":"a67eeecc3860c89f4498de5fa8a792fd0fcd8a4fbc96ed7e84a90b2a41d4aa10"} Oct 30 21:18:37.600994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.600646 5039 scope.go:122] "RemoveContainer" containerID="9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be" Oct 30 21:18:37.607738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.607705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:37.611187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.611170 5039 scope.go:122] "RemoveContainer" containerID="9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be" Oct 30 21:18:37.611460 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:18:37.611431 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be\": container with ID starting with 9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be not found: ID does not exist" containerID="9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be" Oct 30 21:18:37.611506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.611461 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be"} err="failed to get container status \"9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be\": rpc error: code = NotFound desc = could not find container \"9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be\": container with ID starting with 9fd25d11e1cde60a530abbbccb26743c05faac8fbd6d7e3ce4ed9b2520ed25be not found: ID does not exist" Oct 30 21:18:37.646246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.646213 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l"] Oct 30 21:18:37.655797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.655763 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hfx9l"] Oct 30 21:18:37.751722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:37.751687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:38.920921 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:38.920886 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007ac525-f729-4832-ac3c-4d6270af60c6" path="/var/lib/kubelet/pods/007ac525-f729-4832-ac3c-4d6270af60c6/volumes" Oct 30 21:18:40.751458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:40.751420 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:40.894875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:18:40.894840 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:55.184164 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:18:55.184126 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:19:06.461980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:06.461901 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:07.004461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:07.004424 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:08.396275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:08.396238 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:08.541096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:08.541059 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:11.268801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:11.268766 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:12.087097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:12.087065 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:12.250514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:12.250474 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:13.278975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:13.278934 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:19:16.055751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:16.055713 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:17.034454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:17.034415 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:19:18.640322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:18.640285 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:19.858342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:19.858304 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:19:20.960347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:20.960315 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:19:20.961948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:20.961920 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:19:21.864464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:21.864422 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.907683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:22.907647 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:19:23.794743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:23.794702 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:24.226904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.226867 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc"] Oct 30 21:19:24.227423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.227402 5039 cpu_manager.go:414] "containerMap: removing container" podUID="e0f25dba-b014-41c6-ad03-ceb9eba2f777" containerName="maas-api" Oct 30 21:19:24.227499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.227425 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f25dba-b014-41c6-ad03-ceb9eba2f777" containerName="maas-api" Oct 30 21:19:24.227499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.227450 5039 cpu_manager.go:414] "containerMap: removing container" podUID="007ac525-f729-4832-ac3c-4d6270af60c6" containerName="manager" Oct 30 21:19:24.227499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.227459 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007ac525-f729-4832-ac3c-4d6270af60c6" containerName="manager" Oct 30 21:19:24.227640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.227591 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f25dba-b014-41c6-ad03-ceb9eba2f777" containerName="maas-api" Oct 30 21:19:24.227640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.227606 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="007ac525-f729-4832-ac3c-4d6270af60c6" containerName="manager" Oct 30 21:19:24.231989 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.231969 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.245862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.245841 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c2rf5\"" Oct 30 21:19:24.257638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.257609 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc"] Oct 30 21:19:24.315785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.315759 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.315941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.315800 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.315941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.315821 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.316021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.315942 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.316021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.315976 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjq8\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.316084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.316036 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.316084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.316068 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.316147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.316132 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.316179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.316161 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417541 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417594 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417702 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417751 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417797 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjq8\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417820 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417842 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417901 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.417952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417949 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.418192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.417966 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.418192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.418095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.418288 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.418215 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.418288 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.418272 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.418750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.418726 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.420194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.420172 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.420402 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.420369 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.433966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.433936 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.434135 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.434117 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjq8\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8\") pod \"maas-default-gateway-openshift-default-77f556d647-ggsbc\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.545302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.545223 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:24.851532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.851467 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc"] Oct 30 21:19:24.853797 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:19:24.853768 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode652f019_d1d9_4095_a57b_22d4b2016841.slice/crio-d625ce102ca9dbb5b4e79e88123fb36e9553cfa84a65b4b851e7b7cc1fcd3be5 WatchSource:0}: Error finding container d625ce102ca9dbb5b4e79e88123fb36e9553cfa84a65b4b851e7b7cc1fcd3be5: Status 404 returned error can't find the container with id d625ce102ca9dbb5b4e79e88123fb36e9553cfa84a65b4b851e7b7cc1fcd3be5 Oct 30 21:19:24.855977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.855941 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:24.856065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.856024 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:24.856105 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:24.856065 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:25.187049 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:19:25.187010 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:19:25.832006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:25.831963 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" event={"ID":"e652f019-d1d9-4095-a57b-22d4b2016841","Type":"ContainerStarted","Data":"71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d"} Oct 30 21:19:25.832400 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:25.832028 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" event={"ID":"e652f019-d1d9-4095-a57b-22d4b2016841","Type":"ContainerStarted","Data":"d625ce102ca9dbb5b4e79e88123fb36e9553cfa84a65b4b851e7b7cc1fcd3be5"} Oct 30 21:19:25.882416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:25.881751 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podStartSLOduration=1.881730869 podStartE2EDuration="1.881730869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:25.880885024 +0000 UTC m=+605.090824656" watchObservedRunningTime="2026-10-30 21:19:25.881730869 +0000 UTC m=+605.091670499" Oct 30 21:19:26.545873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:26.545836 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:26.547549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:26.547525 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:19:26.547670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:26.547579 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 21:19:27.446060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:27.446010 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:19:27.545950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:27.545914 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:19:27.546123 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:27.545981 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 21:19:28.385416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:28.385032 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:28.546218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:28.546180 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:19:28.546613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:28.546251 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 21:19:29.545607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:29.545571 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:19:29.545790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:29.545631 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 21:19:30.242663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:30.242600 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.546461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:30.546337 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:19:30.546461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:30.546423 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 21:19:31.483166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:31.483124 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:32.587579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:32.587533 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:19:32.588107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:32.587606 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:19:32.832318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:32.832280 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:19:33.059917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:33.059880 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:33.060308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:33.060288 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:33.061048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:33.061030 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:19:39.958598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:39.958542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:40.118494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:40.118459 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:46.072731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:46.072696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:46.225675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:46.225645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:47.641505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:47.641466 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:47.838885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:47.838850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:54.359648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.359607 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs"] Oct 30 21:19:54.363920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.363898 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.390569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.390541 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs"] Oct 30 21:19:54.407689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.407664 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.407842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.407705 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.407842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.407730 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpr9\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.407971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.407875 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.408028 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.407971 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.408083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.408034 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.408083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.408050 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.408186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.408078 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.408186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.408147 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508714 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508682 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508724 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508744 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpr9\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508761 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508777 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508820 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508835 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.508850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508852 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.509174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.508883 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.509174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.509122 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.509174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.509154 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.509318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.509243 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.509318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.509266 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.509593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.509565 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.511180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.511150 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.511409 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.511372 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.524854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.524830 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.525161 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.525132 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpr9\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9\") pod \"maas-default-gateway-openshift-default-77f556d647-26mhs\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.677040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.677007 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:54.980623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.980337 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs"] Oct 30 21:19:54.983061 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:19:54.983033 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a24aabb_824b_4ba2_bfee_aaaae6bd997f.slice/crio-8282d3fb6070b716e0765835b1327e9bc75d4a6b4c57008475390196b1360744 WatchSource:0}: Error finding container 8282d3fb6070b716e0765835b1327e9bc75d4a6b4c57008475390196b1360744: Status 404 returned error can't find the container with id 8282d3fb6070b716e0765835b1327e9bc75d4a6b4c57008475390196b1360744 Oct 30 21:19:54.985088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.985059 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:54.985184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.985117 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:54.985184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:54.985143 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:55.186296 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:19:55.186249 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:19:55.988142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:55.988104 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" event={"ID":"6a24aabb-824b-4ba2-bfee-aaaae6bd997f","Type":"ContainerStarted","Data":"be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d"} Oct 30 21:19:55.988574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:55.988156 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" event={"ID":"6a24aabb-824b-4ba2-bfee-aaaae6bd997f","Type":"ContainerStarted","Data":"8282d3fb6070b716e0765835b1327e9bc75d4a6b4c57008475390196b1360744"} Oct 30 21:19:56.044626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:56.044566 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podStartSLOduration=2.044551576 podStartE2EDuration="2.044551576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:56.034223875 +0000 UTC m=+635.244163505" watchObservedRunningTime="2026-10-30 21:19:56.044551576 +0000 UTC m=+635.254491205" Oct 30 21:19:56.682418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:56.682129 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:19:56.682418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:56.682259 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:19:56.682418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:56.682302 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 21:19:57.677880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:57.677833 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:19:57.678649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:57.678613 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 21:19:58.682417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:58.681807 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:19:58.682417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:58.681869 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 21:19:59.682549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:59.682506 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:19:59.682972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:19:59.682564 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 21:20:00.677830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:00.677789 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:20:00.678007 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:00.677859 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 21:20:01.678364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:01.678321 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:20:01.678881 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:01.678426 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 21:20:02.941134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:02.941098 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:20:02.941626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:02.941421 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:20:02.942352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:02.942332 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:20:05.767061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:05.767023 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:06.013135 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:06.013098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:11.377677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:11.377631 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:14.474743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:14.474707 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:14.646794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:14.646761 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:14.796100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:14.796013 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:14.992615 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:14.992574 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:15.170918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:15.170880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:15.327817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:15.327761 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:15.474146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:15.474044 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:15.716922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:15.716882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:15.870698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:15.870603 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:16.481471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:16.481420 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:16.946564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:16.946528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:17.094220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:17.093917 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:17.839412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:17.839357 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:17.987102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:17.987069 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:18.761157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:18.761128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:18.926891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:18.926852 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:19.324198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.324150 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt"] Oct 30 21:20:19.328128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.328108 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.350542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.350514 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bb245e02-serving-cert\"" Oct 30 21:20:19.350699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.350602 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:20:19.350699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.350599 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q645h\"" Oct 30 21:20:19.352462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.352439 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt"] Oct 30 21:20:19.356125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.356079 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfnq\" (UniqueName: \"kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.356258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.356176 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.356258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.356211 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.356435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.356262 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.457236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.457206 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfnq\" (UniqueName: \"kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.457439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.457266 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.457439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.457287 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.457439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.457315 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.458017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.457980 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.459727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.459708 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.459843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.459787 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.470096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.470068 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:19.482551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.482527 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfnq\" (UniqueName: \"kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq\") pod \"maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.623790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.623696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:19.641540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.641511 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:19.789639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.789601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:19.932351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.932280 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt"] Oct 30 21:20:19.935060 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:20:19.935036 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb2485f_3863_459f_8c40_16303d9f0601.slice/crio-cc09b5d2ac9847d8aa51c4ff4c78d546729fb4e35e6a9176e87a591c9e2f7f2e WatchSource:0}: Error finding container cc09b5d2ac9847d8aa51c4ff4c78d546729fb4e35e6a9176e87a591c9e2f7f2e: Status 404 returned error can't find the container with id cc09b5d2ac9847d8aa51c4ff4c78d546729fb4e35e6a9176e87a591c9e2f7f2e Oct 30 21:20:19.936313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.936298 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:20:19.950424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:19.950383 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:20.114637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:20.114600 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" event={"ID":"3eb2485f-3863-459f-8c40-16303d9f0601","Type":"ContainerStarted","Data":"cc09b5d2ac9847d8aa51c4ff4c78d546729fb4e35e6a9176e87a591c9e2f7f2e"} Oct 30 21:20:20.742077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:20.742046 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:20.890456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:20.890419 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:21.121483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:21.121370 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" event={"ID":"3eb2485f-3863-459f-8c40-16303d9f0601","Type":"ContainerStarted","Data":"253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e"} Oct 30 21:20:21.121483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:21.121478 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:21.157345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:21.157297 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" podStartSLOduration=1.714000969 podStartE2EDuration="2.157284052s" totalImagesPullingTime="443.283083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:21.155810806 +0000 UTC m=+660.365750435" watchObservedRunningTime="2026-10-30 21:20:21.157284052 +0000 UTC m=+660.367223681" Oct 30 21:20:21.305673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:21.305640 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:21.474948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:21.474904 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:21.618276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:21.618232 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:22.119124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.119090 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:20:22.350751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.350715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:22.470659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.470623 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp"] Oct 30 21:20:22.473986 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.473962 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.485829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.485799 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lvz\" (UniqueName: \"kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz\") pod \"payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.485980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.485965 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.488594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.488571 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s"] Oct 30 21:20:22.491963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.491945 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.495831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.495806 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-acb99762-dockercfg-24j89\"" Oct 30 21:20:22.495831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.495816 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-acb99762\"" Oct 30 21:20:22.503435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.503379 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp"] Oct 30 21:20:22.510764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.510737 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:22.518142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.518117 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s"] Oct 30 21:20:22.586956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.586923 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lvz\" (UniqueName: \"kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz\") pod \"payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.587139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.586989 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.587139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.587017 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.587139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.587040 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8qg\" (UniqueName: \"kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg\") pod \"payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.587808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.587787 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.604296 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.604273 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lvz\" (UniqueName: \"kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz\") pod \"payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.688056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.688018 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.688227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.688070 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8qg\" (UniqueName: \"kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg\") pod \"payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.688731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.688697 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.704937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.704910 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8qg\" (UniqueName: \"kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg\") pod \"payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:22.788144 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.788056 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:22.805487 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:22.805458 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:23.089872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.089795 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp"] Oct 30 21:20:23.096809 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:20:23.096779 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc163e3f_27eb_4a74_980d_c26809c4b7d0.slice/crio-0261b7b0cae79b84477dab1c5fc5a9ba992d86afb4e0e9a8cf628f3af6dc9bb7 WatchSource:0}: Error finding container 0261b7b0cae79b84477dab1c5fc5a9ba992d86afb4e0e9a8cf628f3af6dc9bb7: Status 404 returned error can't find the container with id 0261b7b0cae79b84477dab1c5fc5a9ba992d86afb4e0e9a8cf628f3af6dc9bb7 Oct 30 21:20:23.110516 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:20:23.110484 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0301ee15_74b5_4899_a347_9651612d63a5.slice/crio-bdaddd80f1cd752815ba384c57cd67f7e35cc0c641e06fd19a80252fc53de33c WatchSource:0}: Error finding container bdaddd80f1cd752815ba384c57cd67f7e35cc0c641e06fd19a80252fc53de33c: Status 404 returned error can't find the container with id bdaddd80f1cd752815ba384c57cd67f7e35cc0c641e06fd19a80252fc53de33c Oct 30 21:20:23.112921 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.112853 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s"] Oct 30 21:20:23.135987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.135948 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" event={"ID":"0301ee15-74b5-4899-a347-9651612d63a5","Type":"ContainerStarted","Data":"bdaddd80f1cd752815ba384c57cd67f7e35cc0c641e06fd19a80252fc53de33c"} Oct 30 21:20:23.137021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.136999 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" event={"ID":"bc163e3f-27eb-4a74-980d-c26809c4b7d0","Type":"ContainerStarted","Data":"0261b7b0cae79b84477dab1c5fc5a9ba992d86afb4e0e9a8cf628f3af6dc9bb7"} Oct 30 21:20:23.205168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.205139 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:23.372305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.372223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:23.524183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.524150 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:23.663460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:23.663382 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:24.031899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:24.031807 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:24.177302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:24.177259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:25.034497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.034463 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq"] Oct 30 21:20:25.052688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.052660 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.057926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.057897 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq"] Oct 30 21:20:25.124006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.123974 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.124006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.124007 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.124242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.124036 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwm7\" (UniqueName: \"kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.124242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.124072 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.141169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.141139 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b72f7bd8-serving-cert\"" Oct 30 21:20:25.166590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.166550 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:25.185831 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:20:25.185798 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:20:25.225532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.225502 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.225532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.225534 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.225758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.225561 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwm7\" (UniqueName: \"kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.225758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.225593 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.226525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.226491 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.227962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.227936 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.228266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.228062 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.243723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.243680 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwm7\" (UniqueName: \"kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7\") pod \"maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.318283 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.318248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:25.372895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.372871 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:25.681283 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:25.681253 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq"] Oct 30 21:20:25.683418 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:20:25.683367 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe22008_7d64_4d87_a2df_cb4bc17dc7cc.slice/crio-4fbb28e757680a849e63ec3e982b1591eaad94309b70ad2b1a031d2807fadb29 WatchSource:0}: Error finding container 4fbb28e757680a849e63ec3e982b1591eaad94309b70ad2b1a031d2807fadb29: Status 404 returned error can't find the container with id 4fbb28e757680a849e63ec3e982b1591eaad94309b70ad2b1a031d2807fadb29 Oct 30 21:20:26.157973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.157924 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" event={"ID":"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc","Type":"ContainerStarted","Data":"566034f5020930b3cc054b3f3dc598c11afc5f5c53d4864b714074991e1870f9"} Oct 30 21:20:26.158143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.158013 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" event={"ID":"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc","Type":"ContainerStarted","Data":"4fbb28e757680a849e63ec3e982b1591eaad94309b70ad2b1a031d2807fadb29"} Oct 30 21:20:26.158143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.158049 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:26.159624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.159597 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" event={"ID":"0301ee15-74b5-4899-a347-9651612d63a5","Type":"ContainerStarted","Data":"814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be"} Oct 30 21:20:26.159775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.159696 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:26.161251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.161223 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" event={"ID":"bc163e3f-27eb-4a74-980d-c26809c4b7d0","Type":"ContainerStarted","Data":"e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87"} Oct 30 21:20:26.161351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.161273 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:26.194147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.194105 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" podStartSLOduration=0.914199059 podStartE2EDuration="1.194093058s" totalImagesPullingTime="279.893999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:26.192007933 +0000 UTC m=+665.401947564" watchObservedRunningTime="2026-10-30 21:20:26.194093058 +0000 UTC m=+665.404032703" Oct 30 21:20:26.226533 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.226441 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" podStartSLOduration=2.032319761 podStartE2EDuration="4.226425943s" totalImagesPullingTime="2.194106182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:26.225783193 +0000 UTC m=+665.435722822" watchObservedRunningTime="2026-10-30 21:20:26.226425943 +0000 UTC m=+665.436365573" Oct 30 21:20:26.259977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:26.259937 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" podStartSLOduration=1.941009196 podStartE2EDuration="4.259925068s" totalImagesPullingTime="2.318915872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:26.259508025 +0000 UTC m=+665.469447655" watchObservedRunningTime="2026-10-30 21:20:26.259925068 +0000 UTC m=+665.469864714" Oct 30 21:20:27.016701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:27.016663 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:27.135503 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:27.135471 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:27.192204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:27.192162 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:28.783369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:28.783311 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:28.946885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:28.946840 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:29.094454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:29.094343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:30.840197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:30.840154 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:31.750944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:31.750909 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:20:32.173192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:32.173165 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:32.457499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:32.457382 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:34.487561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:34.487521 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:36.542187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.542155 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4"] Oct 30 21:20:36.545724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.545706 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.560629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.560609 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0a3ffe1c-serving-cert\"" Oct 30 21:20:36.569601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.569577 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4"] Oct 30 21:20:36.638451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.638414 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.638632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.638518 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.638632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.638604 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsl8d\" (UniqueName: \"kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.638725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.638696 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.681231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.681202 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:36.740187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.740160 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.740320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.740245 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.740320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.740271 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.740416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.740328 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsl8d\" (UniqueName: \"kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.741026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.740996 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.742518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.742492 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.742613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.742554 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.757418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.757381 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsl8d\" (UniqueName: \"kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d\") pod \"maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:36.836668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.836568 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:36.859331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:36.859304 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:37.149379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:37.149349 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4"] Oct 30 21:20:37.152027 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:20:37.151990 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7602b6e_7b5a_428e_8df5_a4f7c131987e.slice/crio-f18f9bcd75cda366249dcbe1b31cc60eb907ef2d7f86ae68bd047e0469a7b30c WatchSource:0}: Error finding container f18f9bcd75cda366249dcbe1b31cc60eb907ef2d7f86ae68bd047e0469a7b30c: Status 404 returned error can't find the container with id f18f9bcd75cda366249dcbe1b31cc60eb907ef2d7f86ae68bd047e0469a7b30c Oct 30 21:20:37.171097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:37.171067 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:20:37.171260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:37.171235 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:20:37.218425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:37.218371 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" event={"ID":"f7602b6e-7b5a-428e-8df5-a4f7c131987e","Type":"ContainerStarted","Data":"f18f9bcd75cda366249dcbe1b31cc60eb907ef2d7f86ae68bd047e0469a7b30c"} Oct 30 21:20:37.450878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:37.450798 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:20:38.224790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:38.224752 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" event={"ID":"f7602b6e-7b5a-428e-8df5-a4f7c131987e","Type":"ContainerStarted","Data":"fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9"} Oct 30 21:20:38.225192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:38.224816 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:38.260517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:38.260455 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" podStartSLOduration=1.922460695 podStartE2EDuration="2.260441099s" totalImagesPullingTime="337.980404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:38.259862823 +0000 UTC m=+677.469802454" watchObservedRunningTime="2026-10-30 21:20:38.260441099 +0000 UTC m=+677.470380728" Oct 30 21:20:38.547170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:38.547075 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:40.214175 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:40.214135 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:42.079302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:42.079258 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:44.029180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:44.029146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:44.234983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:44.234954 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:20:44.649774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:44.649745 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:44.811801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:44.811758 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:45.117663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:45.117582 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:20:46.059252 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:46.059207 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:46.210979 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:46.210941 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:48.091785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:48.091745 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:49.464968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.464936 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt"] Oct 30 21:20:49.465384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.465164 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" podUID="3eb2485f-3863-459f-8c40-16303d9f0601" containerName="maas-api" containerID="cri-o://253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e" gracePeriod=30 Oct 30 21:20:49.572675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.572639 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:49.729229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.729153 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:49.876162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.876131 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq\" (UniqueName: \"kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq\") pod \"3eb2485f-3863-459f-8c40-16303d9f0601\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " Oct 30 21:20:49.876353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.876179 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls\") pod \"3eb2485f-3863-459f-8c40-16303d9f0601\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " Oct 30 21:20:49.876353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.876202 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs\") pod \"3eb2485f-3863-459f-8c40-16303d9f0601\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " Oct 30 21:20:49.876353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.876237 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca\") pod \"3eb2485f-3863-459f-8c40-16303d9f0601\" (UID: \"3eb2485f-3863-459f-8c40-16303d9f0601\") " Oct 30 21:20:49.876703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.876677 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca" pod "3eb2485f-3863-459f-8c40-16303d9f0601" (UID: "3eb2485f-3863-459f-8c40-16303d9f0601"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:49.878364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.878338 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs" pod "3eb2485f-3863-459f-8c40-16303d9f0601" (UID: "3eb2485f-3863-459f-8c40-16303d9f0601"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:49.878475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.878401 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls" pod "3eb2485f-3863-459f-8c40-16303d9f0601" (UID: "3eb2485f-3863-459f-8c40-16303d9f0601"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:49.878475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.878437 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq" pod "3eb2485f-3863-459f-8c40-16303d9f0601" (UID: "3eb2485f-3863-459f-8c40-16303d9f0601"). InnerVolumeSpecName "kube-api-access-6nfnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:49.977340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.977298 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfnq\" (UniqueName: \"kubernetes.io/projected/3eb2485f-3863-459f-8c40-16303d9f0601-kube-api-access-6nfnq\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:49.977340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.977337 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:49.977592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.977350 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb2485f-3863-459f-8c40-16303d9f0601-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:49.977592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:49.977363 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb2485f-3863-459f-8c40-16303d9f0601-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:50.151142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.151106 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:50.303811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.303774 5039 generic.go:356] "Generic (PLEG): container finished" podID="3eb2485f-3863-459f-8c40-16303d9f0601" containerID="253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e" exitCode=0 Oct 30 21:20:50.303985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.303832 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" Oct 30 21:20:50.303985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.303861 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" event={"ID":"3eb2485f-3863-459f-8c40-16303d9f0601","Type":"ContainerDied","Data":"253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e"} Oct 30 21:20:50.303985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.303933 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt" event={"ID":"3eb2485f-3863-459f-8c40-16303d9f0601","Type":"ContainerDied","Data":"cc09b5d2ac9847d8aa51c4ff4c78d546729fb4e35e6a9176e87a591c9e2f7f2e"} Oct 30 21:20:50.303985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.303964 5039 scope.go:122] "RemoveContainer" containerID="253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e" Oct 30 21:20:50.315230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.315213 5039 scope.go:122] "RemoveContainer" containerID="253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e" Oct 30 21:20:50.315633 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:20:50.315610 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e\": container with ID starting with 253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e not found: ID does not exist" containerID="253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e" Oct 30 21:20:50.315759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.315639 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e"} err="failed to get container status \"253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e\": rpc error: code = NotFound desc = could not find container \"253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e\": container with ID starting with 253762a41d6cc2101c110135d12a310936b043c7ca72cd1e24ce9e7fa8c17a4e not found: ID does not exist" Oct 30 21:20:50.320778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.320750 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:50.347486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.347460 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt"] Oct 30 21:20:50.357281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.357259 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb245e02-78b5fcb55c-44nvt"] Oct 30 21:20:50.482536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.482445 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:50.919995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:50.919960 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb2485f-3863-459f-8c40-16303d9f0601" path="/var/lib/kubelet/pods/3eb2485f-3863-459f-8c40-16303d9f0601/volumes" Oct 30 21:20:51.198278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:51.198183 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:51.343538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:51.343504 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:51.848531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:51.848495 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.080415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:52.080359 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.236668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:52.236639 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:20:52.441576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:52.441532 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.588261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:52.588171 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.738763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:52.738729 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.892137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:52.892107 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:53.438796 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:53.438761 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:53.602965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:53.602932 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:53.749137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:53.749049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:54.079682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.079574 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:54.199520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.199486 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq"] Oct 30 21:20:54.200003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.199739 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" podUID="8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" containerName="maas-api" containerID="cri-o://566034f5020930b3cc054b3f3dc598c11afc5f5c53d4864b714074991e1870f9" gracePeriod=30 Oct 30 21:20:54.246281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.246249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:54.327408 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.327366 5039 generic.go:356] "Generic (PLEG): container finished" podID="8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" containerID="566034f5020930b3cc054b3f3dc598c11afc5f5c53d4864b714074991e1870f9" exitCode=0 Oct 30 21:20:54.327571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.327439 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" event={"ID":"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc","Type":"ContainerDied","Data":"566034f5020930b3cc054b3f3dc598c11afc5f5c53d4864b714074991e1870f9"} Oct 30 21:20:54.398830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.398786 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:54.457235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.457209 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:54.517202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.517164 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca\") pod \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " Oct 30 21:20:54.517202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.517198 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7\" (UniqueName: \"kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7\") pod \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " Oct 30 21:20:54.517481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.517227 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs\") pod \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " Oct 30 21:20:54.517481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.517324 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls\") pod \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\" (UID: \"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc\") " Oct 30 21:20:54.517637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.517600 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca" pod "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" (UID: "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:54.517737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.517718 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:54.519747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.519718 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs" pod "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" (UID: "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:54.519977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.519950 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls" pod "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" (UID: "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:54.520153 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.520137 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7" pod "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" (UID: "8fe22008-7d64-4d87-a2df-cb4bc17dc7cc"). InnerVolumeSpecName "kube-api-access-2jwm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:54.618198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.618163 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwm7\" (UniqueName: \"kubernetes.io/projected/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-kube-api-access-2jwm7\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:54.618198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.618196 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:54.618198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.618206 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:20:54.862619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:54.862586 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:55.050616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.050582 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:55.187566 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:20:55.187530 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:20:55.192148 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.192119 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:55.333658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.333628 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" Oct 30 21:20:55.333828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.333654 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq" event={"ID":"8fe22008-7d64-4d87-a2df-cb4bc17dc7cc","Type":"ContainerDied","Data":"4fbb28e757680a849e63ec3e982b1591eaad94309b70ad2b1a031d2807fadb29"} Oct 30 21:20:55.333828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.333719 5039 scope.go:122] "RemoveContainer" containerID="566034f5020930b3cc054b3f3dc598c11afc5f5c53d4864b714074991e1870f9" Oct 30 21:20:55.380694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.377542 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq"] Oct 30 21:20:55.386012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.385988 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b72f7bd8-7d99b4f477-5cbnq"] Oct 30 21:20:55.446963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.446887 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:55.508001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.507973 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s"] Oct 30 21:20:55.554831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.554795 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp"] Oct 30 21:20:55.606151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.606100 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:55.759764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.759680 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:55.919684 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:55.919650 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:56.679469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:56.679437 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:56.920256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:56.920222 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:56.920537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:56.920518 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" path="/var/lib/kubelet/pods/8fe22008-7d64-4d87-a2df-cb4bc17dc7cc/volumes" Oct 30 21:20:57.108832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.108746 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps"] Oct 30 21:20:57.109429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.109369 5039 cpu_manager.go:414] "containerMap: removing container" podUID="3eb2485f-3863-459f-8c40-16303d9f0601" containerName="maas-api" Oct 30 21:20:57.109429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.109408 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb2485f-3863-459f-8c40-16303d9f0601" containerName="maas-api" Oct 30 21:20:57.109526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.109443 5039 cpu_manager.go:414] "containerMap: removing container" podUID="8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" containerName="maas-api" Oct 30 21:20:57.109526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.109452 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" containerName="maas-api" Oct 30 21:20:57.109601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.109541 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe22008-7d64-4d87-a2df-cb4bc17dc7cc" containerName="maas-api" Oct 30 21:20:57.109601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.109559 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb2485f-3863-459f-8c40-16303d9f0601" containerName="maas-api" Oct 30 21:20:57.117516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.117487 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.140937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.140894 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcfr\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.140947 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141068 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141225 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141130 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141225 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141171 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141225 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141259 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141355 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.141469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.141384 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.150096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.150070 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps"] Oct 30 21:20:57.152649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.152625 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-49738b-openshift-default-dockercfg-h4mdb\"" Oct 30 21:20:57.176931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.176898 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:57.241882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.241850 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.241882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.241884 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.241905 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcfr\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.241925 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.241953 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.241995 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242154 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242237 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242277 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242308 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242362 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242519 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242554 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.242807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.242781 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.244524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.244504 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.244759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.244740 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.263444 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.263409 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.263556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.263495 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcfr\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.392427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.392363 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:57.434439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.434413 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:57.591218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.591185 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:57.740631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.740609 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps"] Oct 30 21:20:57.742797 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:20:57.742772 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b2f6b5_c4bf_46a0_b78e_58994839e3ad.slice/crio-9f5b0540d7a7bfcaf1d2dd0a4f2235e26c028a5468fc4b899a1385a6cd1bafb8 WatchSource:0}: Error finding container 9f5b0540d7a7bfcaf1d2dd0a4f2235e26c028a5468fc4b899a1385a6cd1bafb8: Status 404 returned error can't find the container with id 9f5b0540d7a7bfcaf1d2dd0a4f2235e26c028a5468fc4b899a1385a6cd1bafb8 Oct 30 21:20:57.745031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.745001 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:57.745109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.745066 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:57.745109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.745095 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:57.822631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.822600 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:57.978024 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:57.977985 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:58.121485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.121405 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:58.350746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.350704 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" event={"ID":"80b2f6b5-c4bf-46a0-b78e-58994839e3ad","Type":"ContainerStarted","Data":"3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7"} Oct 30 21:20:58.350897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.350760 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" event={"ID":"80b2f6b5-c4bf-46a0-b78e-58994839e3ad","Type":"ContainerStarted","Data":"9f5b0540d7a7bfcaf1d2dd0a4f2235e26c028a5468fc4b899a1385a6cd1bafb8"} Oct 30 21:20:58.406962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.406917 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podStartSLOduration=1.406902539 podStartE2EDuration="1.406902539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:58.396918248 +0000 UTC m=+697.606857880" watchObservedRunningTime="2026-10-30 21:20:58.406902539 +0000 UTC m=+697.616842169" Oct 30 21:20:58.434848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.434816 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:58.440838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.440813 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:58.624860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.624829 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:58.807962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.807886 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:58.956709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:58.956669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:59.356590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:59.356515 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:59.357683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:59.357651 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:20:59.500908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:59.500875 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:59.673712 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:59.673678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:59.913426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:20:59.913372 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:00.476977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.476936 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:00.508609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.508576 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" podUID="0301ee15-74b5-4899-a347-9651612d63a5" containerName="payload-processing" containerID="cri-o://814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be" gracePeriod=25 Oct 30 21:21:00.555469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.555403 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" podUID="bc163e3f-27eb-4a74-980d-c26809c4b7d0" containerName="payload-pre-processing" containerID="cri-o://e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87" gracePeriod=25 Oct 30 21:21:00.643944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.643910 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:00.836094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.836066 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:21:00.869842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.869817 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:21:00.877457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.877435 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume\") pod \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " Oct 30 21:21:00.877593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.877495 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz\" (UniqueName: \"kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz\") pod \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\" (UID: \"bc163e3f-27eb-4a74-980d-c26809c4b7d0\") " Oct 30 21:21:00.877593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.877547 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg\" (UniqueName: \"kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg\") pod \"0301ee15-74b5-4899-a347-9651612d63a5\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " Oct 30 21:21:00.877707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.877652 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume\") pod \"0301ee15-74b5-4899-a347-9651612d63a5\" (UID: \"0301ee15-74b5-4899-a347-9651612d63a5\") " Oct 30 21:21:00.877827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.877799 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume" pod "bc163e3f-27eb-4a74-980d-c26809c4b7d0" (UID: "bc163e3f-27eb-4a74-980d-c26809c4b7d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:00.878043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.878017 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume" pod "0301ee15-74b5-4899-a347-9651612d63a5" (UID: "0301ee15-74b5-4899-a347-9651612d63a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:00.878147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.878126 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc163e3f-27eb-4a74-980d-c26809c4b7d0-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:00.878147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.878141 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0301ee15-74b5-4899-a347-9651612d63a5-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:00.879656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.879634 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz" pod "bc163e3f-27eb-4a74-980d-c26809c4b7d0" (UID: "bc163e3f-27eb-4a74-980d-c26809c4b7d0"). InnerVolumeSpecName "kube-api-access-s9lvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:00.880118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.880099 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg" pod "0301ee15-74b5-4899-a347-9651612d63a5" (UID: "0301ee15-74b5-4899-a347-9651612d63a5"). InnerVolumeSpecName "kube-api-access-wg8qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:00.979301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.979271 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lvz\" (UniqueName: \"kubernetes.io/projected/bc163e3f-27eb-4a74-980d-c26809c4b7d0-kube-api-access-s9lvz\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:00.979301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:00.979298 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg8qg\" (UniqueName: \"kubernetes.io/projected/0301ee15-74b5-4899-a347-9651612d63a5-kube-api-access-wg8qg\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:01.369651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.369617 5039 generic.go:356] "Generic (PLEG): container finished" podID="bc163e3f-27eb-4a74-980d-c26809c4b7d0" containerID="e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87" exitCode=0 Oct 30 21:21:01.369829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.369673 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" Oct 30 21:21:01.369829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.369698 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" event={"ID":"bc163e3f-27eb-4a74-980d-c26809c4b7d0","Type":"ContainerDied","Data":"e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87"} Oct 30 21:21:01.369829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.369761 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp" event={"ID":"bc163e3f-27eb-4a74-980d-c26809c4b7d0","Type":"ContainerDied","Data":"0261b7b0cae79b84477dab1c5fc5a9ba992d86afb4e0e9a8cf628f3af6dc9bb7"} Oct 30 21:21:01.369829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.369790 5039 scope.go:122] "RemoveContainer" containerID="e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87" Oct 30 21:21:01.371199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.371181 5039 generic.go:356] "Generic (PLEG): container finished" podID="0301ee15-74b5-4899-a347-9651612d63a5" containerID="814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be" exitCode=0 Oct 30 21:21:01.371287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.371213 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" event={"ID":"0301ee15-74b5-4899-a347-9651612d63a5","Type":"ContainerDied","Data":"814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be"} Oct 30 21:21:01.371287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.371234 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" Oct 30 21:21:01.371452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.371284 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s" event={"ID":"0301ee15-74b5-4899-a347-9651612d63a5","Type":"ContainerDied","Data":"bdaddd80f1cd752815ba384c57cd67f7e35cc0c641e06fd19a80252fc53de33c"} Oct 30 21:21:01.378877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.378854 5039 scope.go:122] "RemoveContainer" containerID="e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87" Oct 30 21:21:01.379158 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:21:01.379141 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87\": container with ID starting with e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87 not found: ID does not exist" containerID="e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87" Oct 30 21:21:01.379206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.379167 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87"} err="failed to get container status \"e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87\": rpc error: code = NotFound desc = could not find container \"e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87\": container with ID starting with e2f2b4001cd333fcf8fc88740d638d9c0ae0478d36069d2741b346a9b5d5cf87 not found: ID does not exist" Oct 30 21:21:01.379206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.379185 5039 scope.go:122] "RemoveContainer" containerID="814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be" Oct 30 21:21:01.387121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.387106 5039 scope.go:122] "RemoveContainer" containerID="814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be" Oct 30 21:21:01.387466 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:21:01.387440 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be\": container with ID starting with 814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be not found: ID does not exist" containerID="814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be" Oct 30 21:21:01.387529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.387467 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be"} err="failed to get container status \"814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be\": rpc error: code = NotFound desc = could not find container \"814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be\": container with ID starting with 814d5033740496aed529de3a1bad910d1459db60ac78f4566f9287a656b2a9be not found: ID does not exist" Oct 30 21:21:01.413473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.413451 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp"] Oct 30 21:21:01.419429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.419401 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-acb99762-6bc6dc48ff-dtgwp"] Oct 30 21:21:01.430398 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.430358 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:01.451678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.451654 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s"] Oct 30 21:21:01.456141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.456119 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-acb99762-768fb5c55c-vp49s"] Oct 30 21:21:01.671697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.671649 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:01.856240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:01.856201 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:02.561334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:02.561302 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:02.713574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:02.713535 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:02.920029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:02.919994 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0301ee15-74b5-4899-a347-9651612d63a5" path="/var/lib/kubelet/pods/0301ee15-74b5-4899-a347-9651612d63a5/volumes" Oct 30 21:21:02.920375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:02.920362 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc163e3f-27eb-4a74-980d-c26809c4b7d0" path="/var/lib/kubelet/pods/bc163e3f-27eb-4a74-980d-c26809c4b7d0/volumes" Oct 30 21:21:03.761827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:03.761789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:03.932957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:03.932922 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:04.077828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:04.077743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:04.226087 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:04.226034 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:04.375452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:04.375356 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:05.690944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:05.690911 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:06.972125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:06.972086 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:07.104158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.104127 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4"] Oct 30 21:21:07.104592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.104374 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" podUID="f7602b6e-7b5a-428e-8df5-a4f7c131987e" containerName="maas-api" containerID="cri-o://fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9" gracePeriod=30 Oct 30 21:21:07.127595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.127568 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:07.188181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188150 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2"] Oct 30 21:21:07.188797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188777 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bc163e3f-27eb-4a74-980d-c26809c4b7d0" containerName="payload-pre-processing" Oct 30 21:21:07.188880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188799 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc163e3f-27eb-4a74-980d-c26809c4b7d0" containerName="payload-pre-processing" Oct 30 21:21:07.188880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188849 5039 cpu_manager.go:414] "containerMap: removing container" podUID="0301ee15-74b5-4899-a347-9651612d63a5" containerName="payload-processing" Oct 30 21:21:07.188880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188856 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0301ee15-74b5-4899-a347-9651612d63a5" containerName="payload-processing" Oct 30 21:21:07.188974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188950 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="0301ee15-74b5-4899-a347-9651612d63a5" containerName="payload-processing" Oct 30 21:21:07.188974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.188966 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc163e3f-27eb-4a74-980d-c26809c4b7d0" containerName="payload-pre-processing" Oct 30 21:21:07.192527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.192509 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.213219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.213161 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2"] Oct 30 21:21:07.217055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.217022 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9d677d-dockercfg-4r49b\"" Oct 30 21:21:07.217228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.217054 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9d677d\"" Oct 30 21:21:07.229926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.229866 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.230074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.230047 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkh98\" (UniqueName: \"kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98\") pod \"payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.291258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.291223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:07.331517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.331484 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkh98\" (UniqueName: \"kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98\") pod \"payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.331676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.331573 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.332358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.332309 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.349747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.349718 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkh98\" (UniqueName: \"kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98\") pod \"payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.367928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.367897 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:21:07.406020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.405984 5039 generic.go:356] "Generic (PLEG): container finished" podID="f7602b6e-7b5a-428e-8df5-a4f7c131987e" containerID="fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9" exitCode=0 Oct 30 21:21:07.406201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.406041 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" Oct 30 21:21:07.406201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.406072 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" event={"ID":"f7602b6e-7b5a-428e-8df5-a4f7c131987e","Type":"ContainerDied","Data":"fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9"} Oct 30 21:21:07.406201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.406133 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4" event={"ID":"f7602b6e-7b5a-428e-8df5-a4f7c131987e","Type":"ContainerDied","Data":"f18f9bcd75cda366249dcbe1b31cc60eb907ef2d7f86ae68bd047e0469a7b30c"} Oct 30 21:21:07.406201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.406156 5039 scope.go:122] "RemoveContainer" containerID="fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9" Oct 30 21:21:07.417901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.417878 5039 scope.go:122] "RemoveContainer" containerID="fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9" Oct 30 21:21:07.418193 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:21:07.418174 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9\": container with ID starting with fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9 not found: ID does not exist" containerID="fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9" Oct 30 21:21:07.418264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.418208 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9"} err="failed to get container status \"fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9\": rpc error: code = NotFound desc = could not find container \"fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9\": container with ID starting with fb0b0cc2a679a6e5138bc8c0baa2ff14cac11999d5be498f36a005509d96ece9 not found: ID does not exist" Oct 30 21:21:07.432122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.432102 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca\") pod \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " Oct 30 21:21:07.432226 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.432142 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d\" (UniqueName: \"kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d\") pod \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " Oct 30 21:21:07.432226 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.432215 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls\") pod \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " Oct 30 21:21:07.432306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.432291 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs\") pod \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\" (UID: \"f7602b6e-7b5a-428e-8df5-a4f7c131987e\") " Oct 30 21:21:07.432561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.432535 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca" pod "f7602b6e-7b5a-428e-8df5-a4f7c131987e" (UID: "f7602b6e-7b5a-428e-8df5-a4f7c131987e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:07.434271 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.434242 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d" pod "f7602b6e-7b5a-428e-8df5-a4f7c131987e" (UID: "f7602b6e-7b5a-428e-8df5-a4f7c131987e"). InnerVolumeSpecName "kube-api-access-nsl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:07.434372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.434350 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls" pod "f7602b6e-7b5a-428e-8df5-a4f7c131987e" (UID: "f7602b6e-7b5a-428e-8df5-a4f7c131987e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:07.434465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.434448 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs" pod "f7602b6e-7b5a-428e-8df5-a4f7c131987e" (UID: "f7602b6e-7b5a-428e-8df5-a4f7c131987e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:07.508425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.508338 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:07.533783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.533753 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:07.533883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.533789 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b6e-7b5a-428e-8df5-a4f7c131987e-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:07.533883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.533805 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsl8d\" (UniqueName: \"kubernetes.io/projected/f7602b6e-7b5a-428e-8df5-a4f7c131987e-kube-api-access-nsl8d\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:07.533883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.533820 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7602b6e-7b5a-428e-8df5-a4f7c131987e-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:21:07.616339 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.616302 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:07.760308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.760226 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4"] Oct 30 21:21:07.779481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.779453 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a3ffe1c-5569bb666f-68xk4"] Oct 30 21:21:07.833641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.833615 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2"] Oct 30 21:21:07.834774 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:21:07.834750 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3574dd_52cb_4ab7_85d6_ff35d2b20bc2.slice/crio-7e4e1a2e96819e9cfdd80371b8a5beea47d6beaa888c8da83f7eaf5de45f692d WatchSource:0}: Error finding container 7e4e1a2e96819e9cfdd80371b8a5beea47d6beaa888c8da83f7eaf5de45f692d: Status 404 returned error can't find the container with id 7e4e1a2e96819e9cfdd80371b8a5beea47d6beaa888c8da83f7eaf5de45f692d Oct 30 21:21:07.850552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.850520 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:07.925767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.925738 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796"] Oct 30 21:21:07.926271 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.926252 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f7602b6e-7b5a-428e-8df5-a4f7c131987e" containerName="maas-api" Oct 30 21:21:07.926367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.926273 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7602b6e-7b5a-428e-8df5-a4f7c131987e" containerName="maas-api" Oct 30 21:21:07.926439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.926383 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7602b6e-7b5a-428e-8df5-a4f7c131987e" containerName="maas-api" Oct 30 21:21:07.929796 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.929776 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:07.937092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.937065 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdlvn\" (UniqueName: \"kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:07.937188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.937103 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:07.937188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.937133 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:07.937273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.937245 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:07.942346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.942326 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-49738b-serving-cert\"" Oct 30 21:21:07.942497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.942480 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:21:07.942549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.942503 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q645h\"" Oct 30 21:21:07.951632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:07.951611 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796"] Oct 30 21:21:08.025475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.025378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:08.038719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.038689 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdlvn\" (UniqueName: \"kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.038719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.038721 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.038912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.038756 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.038912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.038786 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.039556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.039534 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.041130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.041110 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.041199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.041141 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.056985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.056957 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdlvn\" (UniqueName: \"kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn\") pod \"maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.173011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.172977 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:08.242330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.242293 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:08.321691 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.321660 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:08.412947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.412911 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" event={"ID":"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2","Type":"ContainerStarted","Data":"29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476"} Oct 30 21:21:08.413145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.412977 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" event={"ID":"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2","Type":"ContainerStarted","Data":"7e4e1a2e96819e9cfdd80371b8a5beea47d6beaa888c8da83f7eaf5de45f692d"} Oct 30 21:21:08.413208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.413138 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:08.447402 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.447335 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" podStartSLOduration=1.447319319 podStartE2EDuration="1.447319319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:08.446280906 +0000 UTC m=+707.656220559" watchObservedRunningTime="2026-10-30 21:21:08.447319319 +0000 UTC m=+707.657258946" Oct 30 21:21:08.532488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.532462 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796"] Oct 30 21:21:08.534919 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:21:08.534885 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed4bc48_36e8_4858_b38e_2f3dbf766df3.slice/crio-2a904d0a47a2a0a6ffddafad98dbcf9f062943763d60f3dcbd4b1032f73a1340 WatchSource:0}: Error finding container 2a904d0a47a2a0a6ffddafad98dbcf9f062943763d60f3dcbd4b1032f73a1340: Status 404 returned error can't find the container with id 2a904d0a47a2a0a6ffddafad98dbcf9f062943763d60f3dcbd4b1032f73a1340 Oct 30 21:21:08.921566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:08.921542 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7602b6e-7b5a-428e-8df5-a4f7c131987e" path="/var/lib/kubelet/pods/f7602b6e-7b5a-428e-8df5-a4f7c131987e/volumes" Oct 30 21:21:09.097836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.097801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:09.421270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.421230 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" event={"ID":"3ed4bc48-36e8-4858-b38e-2f3dbf766df3","Type":"ContainerStarted","Data":"a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5"} Oct 30 21:21:09.421462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.421294 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" event={"ID":"3ed4bc48-36e8-4858-b38e-2f3dbf766df3","Type":"ContainerStarted","Data":"2a904d0a47a2a0a6ffddafad98dbcf9f062943763d60f3dcbd4b1032f73a1340"} Oct 30 21:21:09.421462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.421320 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:09.458584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.458521 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" podStartSLOduration=2.1341680849999998 podStartE2EDuration="2.458502754s" totalImagesPullingTime="324.334669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:09.457828849 +0000 UTC m=+708.667768478" watchObservedRunningTime="2026-10-30 21:21:09.458502754 +0000 UTC m=+708.668442383" Oct 30 21:21:09.574141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.574103 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:09.748743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:09.748640 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:10.124950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:10.124834 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:11.518341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.518305 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:11.734452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.734419 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:11.939568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.939534 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c"] Oct 30 21:21:11.944240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.944216 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:11.959462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.959439 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9d677d-serving-cert\"" Oct 30 21:21:11.968025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.967998 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c"] Oct 30 21:21:11.982070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.982036 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:11.982334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.982308 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:11.982526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.982489 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92p7z\" (UniqueName: \"kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:11.982670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:11.982653 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.083552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.083510 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92p7z\" (UniqueName: \"kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.083552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.083570 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.083888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.083517 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:12.084045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.083904 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.084152 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.084115 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.085045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.085015 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.092794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.092768 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.092974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.092954 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.101089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.101060 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92p7z\" (UniqueName: \"kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z\") pod \"maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.226491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.226378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:12.257302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.257267 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:12.579464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:12.575772 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c"] Oct 30 21:21:12.586240 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:21:12.586209 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36efbae_5b8f_4d22_a6fe_6b460d5c3688.slice/crio-e7a3c79500073b6d8c1eec65ebdaa97aaa22928fa2ebc3110fffa41fef93002e WatchSource:0}: Error finding container e7a3c79500073b6d8c1eec65ebdaa97aaa22928fa2ebc3110fffa41fef93002e: Status 404 returned error can't find the container with id e7a3c79500073b6d8c1eec65ebdaa97aaa22928fa2ebc3110fffa41fef93002e Oct 30 21:21:13.443106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:13.443066 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" event={"ID":"b36efbae-5b8f-4d22-a6fe-6b460d5c3688","Type":"ContainerStarted","Data":"afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92"} Oct 30 21:21:13.443306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:13.443122 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" event={"ID":"b36efbae-5b8f-4d22-a6fe-6b460d5c3688","Type":"ContainerStarted","Data":"e7a3c79500073b6d8c1eec65ebdaa97aaa22928fa2ebc3110fffa41fef93002e"} Oct 30 21:21:13.443306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:13.443150 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:13.491742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:13.491708 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:13.496142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:13.496094 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" podStartSLOduration=2.175847956 podStartE2EDuration="2.496081481s" totalImagesPullingTime="320.233525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:13.49425271 +0000 UTC m=+712.704192352" watchObservedRunningTime="2026-10-30 21:21:13.496081481 +0000 UTC m=+712.706021111" Oct 30 21:21:13.639778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:13.639722 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:14.313029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:14.312987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:15.384030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:15.383975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:15.432584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:15.432555 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 21:21:15.551859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:15.551827 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:16.014907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:16.014868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:16.223997 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:16.223965 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:16.907872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:16.907826 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:17.559125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:17.559075 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:17.733601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:17.733516 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:17.895424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:17.895377 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:18.051380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:18.051238 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:18.342622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:18.342520 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:18.920980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:18.920948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:19.078517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:19.078463 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:19.240159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:19.240069 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:19.425729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:19.425697 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 21:21:19.454369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:19.454340 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 21:21:19.899986 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:19.899942 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:20.485236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:20.485159 5039 generic.go:356] "Generic (PLEG): container finished" podID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerID="3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7" exitCode=0 Oct 30 21:21:20.485236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:20.485206 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" event={"ID":"80b2f6b5-c4bf-46a0-b78e-58994839e3ad","Type":"ContainerDied","Data":"3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7"} Oct 30 21:21:20.485709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:20.485590 5039 scope.go:122] "RemoveContainer" containerID="3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7" Oct 30 21:21:20.486880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:20.486838 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:20.486982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:20.486914 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:20.486982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:20.486968 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:21.100192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.100160 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6"] Oct 30 21:21:21.105206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.105178 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.122935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.122911 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"openshift-service-ca.crt\"" Oct 30 21:21:21.129822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.129796 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"kube-root-ca.crt\"" Oct 30 21:21:21.131991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.131961 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6"] Oct 30 21:21:21.132803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.132787 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"default-dockercfg-vrsxp\"" Oct 30 21:21:21.132872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.132784 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"simulator-w4-ea40ee-kserve-self-signed-certs\"" Oct 30 21:21:21.189726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.189686 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.190017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.189989 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.190147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.190089 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.190215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.190157 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.190215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.190183 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m67vx\" (UniqueName: \"kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.190344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.190274 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.190514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.190482 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:21.293477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.292922 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.293740 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.293810 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.293893 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.294277 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.294354 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.294673 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.294939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.294904 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.295071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.294973 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m67vx\" (UniqueName: \"kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.298380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.297659 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.298380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.298261 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.311795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.311763 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m67vx\" (UniqueName: \"kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx\") pod \"simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.338198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.338150 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:21.420192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.420154 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:21.497050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.497016 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" event={"ID":"80b2f6b5-c4bf-46a0-b78e-58994839e3ad","Type":"ContainerStarted","Data":"9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8"} Oct 30 21:21:21.587936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.587777 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:21.752327 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.752297 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6"] Oct 30 21:21:21.754155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.754116 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:21.754688 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:21:21.754665 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod845dd177_f965_4e53_9463_d35664bbc4a3.slice/crio-e70365c7a76ed5edaf5dc3ec95fa5c4cee6a0c63103fa72ef0a703b77179ad56 WatchSource:0}: Error finding container e70365c7a76ed5edaf5dc3ec95fa5c4cee6a0c63103fa72ef0a703b77179ad56: Status 404 returned error can't find the container with id e70365c7a76ed5edaf5dc3ec95fa5c4cee6a0c63103fa72ef0a703b77179ad56 Oct 30 21:21:21.920555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:21.920514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:22.094665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:22.094610 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:22.287880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:22.283850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:22.434789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:22.434761 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:21:22.434957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:22.434859 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:21:22.434957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:22.434915 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:21:22.506226 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:22.506188 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" event={"ID":"845dd177-f965-4e53-9463-d35664bbc4a3","Type":"ContainerStarted","Data":"e70365c7a76ed5edaf5dc3ec95fa5c4cee6a0c63103fa72ef0a703b77179ad56"} Oct 30 21:21:23.395537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:23.395501 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:23.434996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:23.434974 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:21:23.435110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:23.435018 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:21:23.514290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:23.514251 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" event={"ID":"845dd177-f965-4e53-9463-d35664bbc4a3","Type":"ContainerStarted","Data":"b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334"} Oct 30 21:21:23.550968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:23.550897 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podStartSLOduration=0.957485459 podStartE2EDuration="2.550862624s" totalImagesPullingTime="1.593377165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:23.548488114 +0000 UTC m=+722.758427744" watchObservedRunningTime="2026-10-30 21:21:23.550862624 +0000 UTC m=+722.760802255" Oct 30 21:21:24.421276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:24.421242 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:24.434815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:24.434785 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:21:24.434944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:24.434830 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:21:24.934980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:24.934943 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:25.051160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:25.051122 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:25.218210 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:21:25.218125 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:21:25.434955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:25.434917 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:21:25.435119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:25.434974 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 21:21:26.452458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:26.452422 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:21:26.452932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:26.452764 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:21:26.453822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:26.453796 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 21:21:27.024282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:27.024248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:28.267174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:28.267136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:28.619731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:28.619644 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:28.874502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:28.874412 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.030769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:29.030738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:29.177334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:29.177297 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:30.246150 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.246103 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296"] Oct 30 21:21:30.250382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.250350 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.263185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.263163 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"kube-root-ca.crt\"" Oct 30 21:21:30.272799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.272759 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"openshift-service-ca.crt\"" Oct 30 21:21:30.275176 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.275145 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296"] Oct 30 21:21:30.286112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.286089 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.286217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.286138 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgp8l\" (UniqueName: \"kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.286302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.286277 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.286352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.286335 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.286476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.286459 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.286527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.286495 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.297884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.297868 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"default-dockercfg-ktmlc\"" Oct 30 21:21:30.297964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.297926 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"simulator-w3-50ab77-kserve-self-signed-certs\"" Oct 30 21:21:30.364374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.364339 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:30.387592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.387564 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.387795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.387607 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.387795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.387666 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.387795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.387690 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.387795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.387734 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.387795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.387779 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgp8l\" (UniqueName: \"kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.388078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.388056 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.388138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.388088 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.388419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.388362 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.390369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.390340 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.390882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.390860 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.405830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.405803 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgp8l\" (UniqueName: \"kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l\") pod \"simulator-w3-50ab77-kserve-bb75d44d6-br296\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.506534 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.506430 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:30.565929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.565891 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:30.859799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:30.859768 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296"] Oct 30 21:21:30.861124 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:21:30.861100 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93938b5f_db5a_4a74_b3a3_7c7fd42c8362.slice/crio-06bf908f4149fecc36f19dabb6473a611f85fcfceecaefac594d297e2a5a84de WatchSource:0}: Error finding container 06bf908f4149fecc36f19dabb6473a611f85fcfceecaefac594d297e2a5a84de: Status 404 returned error can't find the container with id 06bf908f4149fecc36f19dabb6473a611f85fcfceecaefac594d297e2a5a84de Oct 30 21:21:31.420343 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.420307 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:31.432777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.432752 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:31.556059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.556022 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" event={"ID":"93938b5f-db5a-4a74-b3a3-7c7fd42c8362","Type":"ContainerStarted","Data":"ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb"} Oct 30 21:21:31.556059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.556071 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" event={"ID":"93938b5f-db5a-4a74-b3a3-7c7fd42c8362","Type":"ContainerStarted","Data":"06bf908f4149fecc36f19dabb6473a611f85fcfceecaefac594d297e2a5a84de"} Oct 30 21:21:31.566675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.566651 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:31.573478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.573454 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:21:31.590780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:31.590737 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podStartSLOduration=1.590725564 podStartE2EDuration="1.590725564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:31.589686359 +0000 UTC m=+730.799625990" watchObservedRunningTime="2026-10-30 21:21:31.590725564 +0000 UTC m=+730.800665194" Oct 30 21:21:32.155909 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:32.155845 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:32.313435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:32.313377 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:33.792680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:33.792630 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:34.358109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:34.358076 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:35.609874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:35.609839 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:37.413805 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:37.413769 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.090169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:38.090136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.702000 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:38.701957 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.846485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:38.846453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.986118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:38.986030 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:39.151203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:39.151164 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:40.424858 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:40.424817 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:40.566963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:40.566931 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:40.583137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:40.583080 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:40.615004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:40.614978 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:21:40.930831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:40.930798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:42.595205 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:42.595163 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:44.203779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:44.203738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:44.329738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:44.329706 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:45.801103 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:45.801057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:47.389064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:47.389028 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:48.899153 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:48.899112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:50.393922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.393887 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v"] Oct 30 21:21:50.400801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.400774 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.412920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.412893 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"kube-root-ca.crt\"" Oct 30 21:21:50.422449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.422424 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"openshift-service-ca.crt\"" Oct 30 21:21:50.422449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.422434 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"default-dockercfg-tn85b\"" Oct 30 21:21:50.422629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.422477 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"simulator-w1-b31518-kserve-self-signed-certs\"" Oct 30 21:21:50.425358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.425335 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v"] Oct 30 21:21:50.484047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.484008 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.484047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.484044 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.484288 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.484077 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.484288 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.484249 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.484423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.484287 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthgt\" (UniqueName: \"kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.484423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.484311 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.495036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.495008 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:50.585751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.585711 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.585751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.585762 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.585817 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.585921 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.585958 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthgt\" (UniqueName: \"kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.585992 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.586147 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.586216 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.586357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.586337 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.588155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.588122 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.588365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.588346 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.602299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.602273 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthgt\" (UniqueName: \"kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt\") pod \"simulator-w1-b31518-kserve-6b8584c848-f7w4v\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:50.649223 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.649107 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:50.718442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:50.718386 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:51.013722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:51.013690 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v"] Oct 30 21:21:51.015669 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:21:51.015640 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34a22871_a835_4685_8adc_ebe3e1960c6f.slice/crio-c1bb9eeaac976c1de6567c9167e7e8ce533b50d0b0073a1f7f1e4f1b2593bf6f WatchSource:0}: Error finding container c1bb9eeaac976c1de6567c9167e7e8ce533b50d0b0073a1f7f1e4f1b2593bf6f: Status 404 returned error can't find the container with id c1bb9eeaac976c1de6567c9167e7e8ce533b50d0b0073a1f7f1e4f1b2593bf6f Oct 30 21:21:51.663840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:51.663801 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" event={"ID":"34a22871-a835-4685-8adc-ebe3e1960c6f","Type":"ContainerStarted","Data":"fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6"} Oct 30 21:21:51.664260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:51.663849 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" event={"ID":"34a22871-a835-4685-8adc-ebe3e1960c6f","Type":"ContainerStarted","Data":"c1bb9eeaac976c1de6567c9167e7e8ce533b50d0b0073a1f7f1e4f1b2593bf6f"} Oct 30 21:21:51.700637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:51.700592 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podStartSLOduration=1.700578653 podStartE2EDuration="1.700578653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:51.699333447 +0000 UTC m=+750.909273076" watchObservedRunningTime="2026-10-30 21:21:51.700578653 +0000 UTC m=+750.910518282" Oct 30 21:21:51.718727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:51.718700 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:21:52.204526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:52.204490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:53.802311 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:53.802269 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:53.964317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:53.964276 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:55.183939 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:21:55.183904 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:21:55.316563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:55.316526 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:56.340680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:56.340643 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:21:56.892855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:56.892815 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:58.217273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:58.217234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:58.610170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:21:58.610057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:00.198380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:00.198341 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:00.719240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:00.719204 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:22:00.731602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:00.731574 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:22:01.729527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:01.729497 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:22:01.794876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:01.794834 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:03.287982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:03.287896 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:04.909863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:04.909814 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:06.601559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.601522 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:06.722873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.722834 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h"] Oct 30 21:22:06.727112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.727082 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.755593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.755558 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h"] Oct 30 21:22:06.755749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.755593 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"premium-w3-50ab77-kserve-self-signed-certs\"" Oct 30 21:22:06.832610 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.832571 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:06.850790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.850763 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.850962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.850808 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.850962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.850839 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.850962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.850950 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.851134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.850987 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65q5\" (UniqueName: \"kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.851134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.851121 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.951906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.951859 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.952143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.951979 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.952143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.952040 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.952143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.952089 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.952330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.952167 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.952330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.952212 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65q5\" (UniqueName: \"kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.952982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.952952 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.953673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.953497 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.953855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.953612 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.955746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.955722 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.962419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.962284 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:06.969215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:06.969190 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65q5\" (UniqueName: \"kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5\") pod \"premium-w3-50ab77-kserve-85d7954875-mhx4h\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:07.041215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:07.041183 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:07.332622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:07.332596 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h"] Oct 30 21:22:07.334606 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:22:07.334580 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod436b8ce8_13a5_4b79_bd40_868bfb93947b.slice/crio-e1448dd4b343ff55fcb4f0125f91f144635c9a1fe0ab2724f4d22fd04a3df927 WatchSource:0}: Error finding container e1448dd4b343ff55fcb4f0125f91f144635c9a1fe0ab2724f4d22fd04a3df927: Status 404 returned error can't find the container with id e1448dd4b343ff55fcb4f0125f91f144635c9a1fe0ab2724f4d22fd04a3df927 Oct 30 21:22:07.750648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:07.750609 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" event={"ID":"436b8ce8-13a5-4b79-bd40-868bfb93947b","Type":"ContainerStarted","Data":"4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853"} Oct 30 21:22:07.751029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:07.750655 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" event={"ID":"436b8ce8-13a5-4b79-bd40-868bfb93947b","Type":"ContainerStarted","Data":"e1448dd4b343ff55fcb4f0125f91f144635c9a1fe0ab2724f4d22fd04a3df927"} Oct 30 21:22:07.786720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:07.786661 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podStartSLOduration=1.786646588 podStartE2EDuration="1.786646588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:07.785341182 +0000 UTC m=+766.995280813" watchObservedRunningTime="2026-10-30 21:22:07.786646588 +0000 UTC m=+766.996586218" Oct 30 21:22:08.042123 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:08.042047 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:08.108722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:08.108686 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:09.659279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:09.659232 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:09.738868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:09.738827 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:22:11.286679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:11.286641 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:11.621054 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:11.620963 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:12.984611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:12.984547 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:13.647116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.647079 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq"] Oct 30 21:22:13.650829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.650804 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.668728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.668702 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq"] Oct 30 21:22:13.678333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.678305 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"openshift-service-ca.crt\"" Oct 30 21:22:13.678527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.678473 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"kube-root-ca.crt\"" Oct 30 21:22:13.679159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.679121 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"premium-w2-9d677d-kserve-self-signed-certs\"" Oct 30 21:22:13.679263 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.679183 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"default-dockercfg-dxblp\"" Oct 30 21:22:13.720273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.720232 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.720273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.720263 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.720497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.720282 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.720497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.720375 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w266l\" (UniqueName: \"kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.720497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.720432 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.720658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.720538 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.755435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.755374 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:13.821531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.821494 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.821771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.821632 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.821771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.821664 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.821771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.821691 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.821771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.821743 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w266l\" (UniqueName: \"kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.821921 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.821789 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.822478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.822454 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.822934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.822899 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.823134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.823114 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.824611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.824565 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.824737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.824711 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.852428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.851086 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w266l\" (UniqueName: \"kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l\") pod \"premium-w2-9d677d-kserve-6b74745659-ddrjq\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:13.966961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:13.966879 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:14.258811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:14.258781 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq"] Oct 30 21:22:14.260659 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:22:14.260631 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf6be14_5c6a_4657_ae5d_013b45d9f1d3.slice/crio-a362648469fb6cdb1f919f9bf17e4b78315e13ac72668a3d12e67828c2db092a WatchSource:0}: Error finding container a362648469fb6cdb1f919f9bf17e4b78315e13ac72668a3d12e67828c2db092a: Status 404 returned error can't find the container with id a362648469fb6cdb1f919f9bf17e4b78315e13ac72668a3d12e67828c2db092a Oct 30 21:22:14.652915 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:14.652878 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:14.790317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:14.790278 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" event={"ID":"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3","Type":"ContainerStarted","Data":"9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b"} Oct 30 21:22:14.790317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:14.790320 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" event={"ID":"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3","Type":"ContainerStarted","Data":"a362648469fb6cdb1f919f9bf17e4b78315e13ac72668a3d12e67828c2db092a"} Oct 30 21:22:14.826678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:14.826595 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podStartSLOduration=1.82658006 podStartE2EDuration="1.82658006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:14.824665729 +0000 UTC m=+774.034605360" watchObservedRunningTime="2026-10-30 21:22:14.82658006 +0000 UTC m=+774.036519722" Oct 30 21:22:14.967052 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:14.966957 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:16.201507 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:16.201469 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:17.041655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:17.041618 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:17.053953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:17.053925 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:17.817124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:17.817092 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:22:17.900117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:17.900085 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:19.593585 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:19.593549 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:20.722267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:20.722223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:20.859096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:20.859059 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:21.290174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:21.290130 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:22.874797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:22.874766 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:23.967467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:23.967427 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:23.981958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:23.981929 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:24.390133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:24.390085 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:24.548650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:24.548604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:24.858147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:24.858064 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:22:25.184329 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:22:25.184282 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:22:26.095121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:26.095077 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:27.690378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:27.690337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:28.832527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:28.832497 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:22:29.287847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:29.287805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:31.031445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:31.031357 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:32.692450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:32.692415 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:33.497171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:33.497136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:34.289990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:34.289953 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:35.787167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:35.787126 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:37.382077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:37.382016 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:38.969632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:38.969597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:40.828530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:40.828493 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:42.432064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:42.432026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:44.217656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:44.217605 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:46.005302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:46.005261 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:46.745687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:46.745657 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:22:47.619790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:47.619751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:49.443619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:49.443584 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:51.051768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:51.051727 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:52.614817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:52.614783 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:54.243036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:54.242991 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:55.185542 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:22:55.185505 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:22:55.800992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:55.800958 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:57.722651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:57.722609 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:59.294104 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:22:59.294062 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:00.976145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:00.976113 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:02.592974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:02.592932 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:03.403107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:03.403071 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:23:04.321174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:04.321140 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:05.903145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:05.903108 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:07.420031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:07.419987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:08.402300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:08.402256 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:08.542685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:08.542656 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:09.005844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:09.005813 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:10.608110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:10.607918 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:11.738283 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:11.738243 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:12.044559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:12.044445 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:12.178431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:12.178381 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:12.319363 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:12.319272 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:13.707910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:13.707867 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:13.991893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:13.991811 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8"] Oct 30 21:23:13.995539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:13.995519 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.025378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.025357 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 21:23:14.025682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.025661 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 21:23:14.026124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.026092 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tlp\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.026242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.026224 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.026292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.026260 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.026370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.026355 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.037841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.037795 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8"] Oct 30 21:23:14.038182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.038167 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-18092bbdb3eb\"" Oct 30 21:23:14.038540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.038522 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d93cf183177\"" Oct 30 21:23:14.041297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.041280 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dmwj5\"" Oct 30 21:23:14.095650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.095629 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 21:23:14.095805 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.095787 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 21:23:14.127139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.127116 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.127269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.127147 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.127269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.127186 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.127376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.127284 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tlp\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.127972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.127950 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.127972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.127967 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.129679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.129655 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.145734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.145711 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tlp\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp\") pod \"payload-processing-external-model-5c46ddcfbd-8xjq8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.274077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.273935 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:14.308782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.308744 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:14.617461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:14.617294 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8"] Oct 30 21:23:14.619589 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:23:14.619560 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca512b1f_81ec_4f41_9178_d78794fd01c8.slice/crio-cf6f07f10c7134a8b7c623f39c32e47a4802c03c165d7d317c0151d0ee0e52bb WatchSource:0}: Error finding container cf6f07f10c7134a8b7c623f39c32e47a4802c03c165d7d317c0151d0ee0e52bb: Status 404 returned error can't find the container with id cf6f07f10c7134a8b7c623f39c32e47a4802c03c165d7d317c0151d0ee0e52bb Oct 30 21:23:15.114709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:15.114671 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" event={"ID":"ca512b1f-81ec-4f41-9178-d78794fd01c8","Type":"ContainerStarted","Data":"118d314a39abd6dc0a4ff2d64827dedef97ef1a4f726a0883f58b49c3e008eda"} Oct 30 21:23:15.115115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:15.114722 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" event={"ID":"ca512b1f-81ec-4f41-9178-d78794fd01c8","Type":"ContainerStarted","Data":"cf6f07f10c7134a8b7c623f39c32e47a4802c03c165d7d317c0151d0ee0e52bb"} Oct 30 21:23:15.115115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:15.114774 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:15.147799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:15.147743 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" podStartSLOduration=2.147728412 podStartE2EDuration="2.147728412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:15.146924578 +0000 UTC m=+834.356864209" watchObservedRunningTime="2026-10-30 21:23:15.147728412 +0000 UTC m=+834.357668041" Oct 30 21:23:16.102568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:16.101520 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:17.812575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:17.812538 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:19.393003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:19.392962 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:20.883187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:20.883149 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:21.654671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.654633 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q"] Oct 30 21:23:21.658742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.658719 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.675625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.675598 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q"] Oct 30 21:23:21.682553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.682521 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"e2e-conflict-a-26510fbf-kserve-self-signed-certs\"" Oct 30 21:23:21.707469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.707443 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.707594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.707483 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.707594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.707510 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.707664 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.707629 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lkc\" (UniqueName: \"kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.707702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.707680 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.707778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.707753 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.762777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.762741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:21.808593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.808564 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.808774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.808607 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.808774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.808638 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.808774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.808684 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lkc\" (UniqueName: \"kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.808774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.808748 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.808987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.808780 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.809161 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.809138 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.809744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.809718 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.809864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.809774 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.811427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.811405 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.811983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.811958 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.825172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.825148 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lkc\" (UniqueName: \"kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc\") pod \"e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:21.978372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:21.978286 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:22.270059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:22.270033 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q"] Oct 30 21:23:22.498535 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:22.498498 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:23.142091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.142054 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:23:23.164748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.164714 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" event={"ID":"7bc5635e-40db-4b80-a5f0-58274c247fb8","Type":"ContainerStarted","Data":"a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661"} Oct 30 21:23:23.164934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.164757 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" event={"ID":"7bc5635e-40db-4b80-a5f0-58274c247fb8","Type":"ContainerStarted","Data":"7898ae1d099b669b1b64db3151263d2f8690975d02c09b357b6435abc5a501b2"} Oct 30 21:23:23.200974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.200922 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" podStartSLOduration=2.200909987 podStartE2EDuration="2.200909987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:23.200651476 +0000 UTC m=+842.410591105" watchObservedRunningTime="2026-10-30 21:23:23.200909987 +0000 UTC m=+842.410849617" Oct 30 21:23:23.796283 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.796244 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x"] Oct 30 21:23:23.800402 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.800365 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.814442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.814420 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea40ee-openshift-default-dockercfg-8mf6h\"" Oct 30 21:23:23.827556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.827522 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x"] Oct 30 21:23:23.933270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933223 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933287 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933438 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933502 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933597 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933633 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933686 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjxq\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933825 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933721 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.933825 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.933758 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:23.986414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:23.985484 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:24.035403 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035331 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035429 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035476 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035531 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035572 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035667 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035704 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035761 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjxq\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035799 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.035902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035880 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.036163 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.035954 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.036227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.036177 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.036722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.036694 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.036824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.036803 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.038621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.038591 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.040965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.040942 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.054236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.054177 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.054789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.054763 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjxq\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.076432 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.076202 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:24.116584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.116270 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:24.418145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.415918 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls"] Oct 30 21:23:24.421624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.421598 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.443317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.443290 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls"] Oct 30 21:23:24.446657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.446633 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-50ab77-openshift-default-dockercfg-tt79b\"" Oct 30 21:23:24.471793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.471761 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x"] Oct 30 21:23:24.472035 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:23:24.472011 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dbc5be_53dd_430e_ab1b_bea41e346093.slice/crio-fa60aa986a19bdb5dc5d511770c01a597cba552eddec8cf34061bbd3e4e657e4 WatchSource:0}: Error finding container fa60aa986a19bdb5dc5d511770c01a597cba552eddec8cf34061bbd3e4e657e4: Status 404 returned error can't find the container with id fa60aa986a19bdb5dc5d511770c01a597cba552eddec8cf34061bbd3e4e657e4 Oct 30 21:23:24.474996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.474951 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:24.475103 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.475035 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:24.475174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.475109 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:24.541821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541739 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.541821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541782 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.541821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541814 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.542016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541853 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.542016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541895 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.542016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541959 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.542016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.541993 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmtc\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.542244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.542217 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.542321 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.542264 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643646 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643706 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmtc\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643761 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643789 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643893 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643927 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.643966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.643962 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.644247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.644002 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.644247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.644048 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.644366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.644336 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.644485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.644460 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.644554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.644466 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.644983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.644702 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.645229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.645204 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.646971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.646944 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.647292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.647267 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.660632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.660605 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmtc\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.661490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.661464 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.741053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.740959 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:24.923438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.923380 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf"] Oct 30 21:23:24.928437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.928413 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:24.980415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:24.963440 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf"] Oct 30 21:23:25.050682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050564 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.050682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050614 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjj27\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.050682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050649 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.050682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050680 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.051005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050721 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.051005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050743 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.051005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050793 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.051005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050862 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.051005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.050939 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.108118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.107890 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls"] Oct 30 21:23:25.109420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.108991 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:25.109420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.109076 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:25.109420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.109127 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152056 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152164 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152195 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjj27\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152229 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152264 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152304 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.152331 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.153291 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.153957 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.154484 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.155315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.154877 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.159380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.156059 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.159380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.158019 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.159380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.158441 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.160911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.160848 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.166517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.166239 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.191991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.191879 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" event={"ID":"0a3e1041-a476-42d7-bb2c-d5d8b813161f","Type":"ContainerStarted","Data":"e72b74d37e832b3e285fa82426ba568bfc9ea09e6109672920f3100b57aa45ed"} Oct 30 21:23:25.206843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.206805 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" event={"ID":"c8dbc5be-53dd-430e-ab1b-bea41e346093","Type":"ContainerStarted","Data":"4b5052c7d949844af315471b3e15372467ae68c6d3c91d99e1bfdd7fe2ef695e"} Oct 30 21:23:25.207158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.207135 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" event={"ID":"c8dbc5be-53dd-430e-ab1b-bea41e346093","Type":"ContainerStarted","Data":"fa60aa986a19bdb5dc5d511770c01a597cba552eddec8cf34061bbd3e4e657e4"} Oct 30 21:23:25.213257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.213196 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.214405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.214042 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjj27\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27\") pod \"maas-default-gateway-openshift-default-77f556d647-lbmvf\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.250058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:25.249755 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:25.324466 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:23:25.287929 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:23:26.038425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.034658 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podStartSLOduration=3.034637237 podStartE2EDuration="3.034637237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:25.250787534 +0000 UTC m=+844.460727176" watchObservedRunningTime="2026-10-30 21:23:26.034637237 +0000 UTC m=+845.244576865" Oct 30 21:23:26.054206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.054174 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf"] Oct 30 21:23:26.054557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.054521 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:26.054761 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.054739 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:26.054885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.054869 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:26.118758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.118709 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:26.120954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.120844 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:23:26.121188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.120920 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:23:26.131644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.131624 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:23:26.338016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.337911 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" event={"ID":"0a3e1041-a476-42d7-bb2c-d5d8b813161f","Type":"ContainerStarted","Data":"ac44dce38d3a9f464d50993b0bcc7da1e41188034dd57728b7b159d6e582709b"} Oct 30 21:23:26.340520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.340449 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" event={"ID":"a7214320-9109-4afe-ab03-0ae40f5861ea","Type":"ContainerStarted","Data":"474dfbb60c804ea0e01982de753146823d5f78296cef556e45c30aafc1fa4f27"} Oct 30 21:23:26.417763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.417700 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podStartSLOduration=2.417678426 podStartE2EDuration="2.417678426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:26.397898911 +0000 UTC m=+845.607838543" watchObservedRunningTime="2026-10-30 21:23:26.417678426 +0000 UTC m=+845.627618056" Oct 30 21:23:26.521440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.506594 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:26.741758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.741725 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:26.745505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.745478 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:23:26.745700 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:26.745674 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:23:27.117304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:27.117212 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:23:27.117304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:27.117271 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:23:27.348978 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:27.348945 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" event={"ID":"a7214320-9109-4afe-ab03-0ae40f5861ea","Type":"ContainerStarted","Data":"df20040c1260416969cb9eb88b1842ef3a0f4231cbff20f5d0b28196a60771ea"} Oct 30 21:23:27.741683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:27.741645 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:23:27.741908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:27.741707 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:23:28.132471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.131832 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:23:28.132471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.131909 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:23:28.251634 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.251596 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:28.253105 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.253080 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:28.253164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.253138 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:28.447027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.446993 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:28.741632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.741542 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:23:28.741885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:28.741839 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:23:29.121490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:29.120927 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:23:29.121490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:29.121002 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:23:29.252097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:29.252046 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:29.252513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:29.252113 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:29.741300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:29.741265 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:23:29.741503 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:29.741327 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:23:30.120472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.119880 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:23:30.120472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.119947 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:23:30.252350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.252309 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:30.252789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.252366 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:30.354905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.354869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:30.742109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.742059 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:23:30.742284 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:30.742112 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:23:31.117501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.117346 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:23:31.117501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.117472 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:23:31.251783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.251751 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:31.251941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.251799 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:31.741904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.741867 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:23:31.742311 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.741922 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:23:31.989573 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:31.989509 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:32.009354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.009263 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:32.044680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.044617 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podStartSLOduration=8.04459813 podStartE2EDuration="8.04459813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:27.400647337 +0000 UTC m=+846.610586989" watchObservedRunningTime="2026-10-30 21:23:32.04459813 +0000 UTC m=+851.254537760" Oct 30 21:23:32.128453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.128417 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:32.128860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.128841 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:32.129861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.129834 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:23:32.252859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.252823 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:32.253114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.253086 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:32.445699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.445668 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:23:32.573501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:32.573462 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:33.198414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:33.195015 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:33.198414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:33.196408 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:33.201442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:33.200676 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:23:33.251931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:33.251897 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:33.252101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:33.251949 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:34.252047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:34.252011 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:23:34.252487 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:34.252065 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 21:23:35.114472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:35.114433 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:36.252088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:36.252043 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:23:36.252655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:36.252117 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:23:37.294652 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:37.294593 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lbmvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:23:37.295307 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:37.295265 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:23:37.308350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:37.308306 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:37.921932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:37.921893 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:37.922310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:37.922280 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:37.923106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:37.923087 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:23:39.079168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:39.079123 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:40.240857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:40.240823 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:23:40.714446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:40.714404 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:40.914557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:40.914514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:42.507170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:42.507126 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:44.216677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:44.216639 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:45.871792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:45.871751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:46.649637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:46.649597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:47.546272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:47.546217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:49.210743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:49.210705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:50.781173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:50.781135 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:54.014411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:54.011819 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:55.184491 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:23:55.184447 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:23:56.272511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:56.272466 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:57.780724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:57.780670 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:59.288779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:23:59.288740 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:00.995925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:00.995885 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:02.584261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:02.584222 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:03.437873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:03.437836 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:24:04.101704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:04.101653 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:05.242350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.242313 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw"] Oct 30 21:24:05.248737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.248715 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.270840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.270816 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"distinct-w1-b31518-kserve-self-signed-certs\"" Oct 30 21:24:05.273277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.273253 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw"] Oct 30 21:24:05.348797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.348758 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:05.349161 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.349139 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.349224 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.349204 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.349304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.349287 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.349345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.349330 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.349505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.349491 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.349558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.349515 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wt8\" (UniqueName: \"kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.451981 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.451944 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.452184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.451996 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wt8\" (UniqueName: \"kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.452184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.452035 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.452184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.452106 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.452184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.452144 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.452407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.452190 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.452875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.452848 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.453369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.453351 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.453698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.453666 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.461446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.456465 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.461446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.461220 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.473430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.473365 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wt8\" (UniqueName: \"kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8\") pod \"distinct-w1-b31518-kserve-fcc7674d4-z2wkw\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.563174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.563082 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:05.733211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.733170 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:05.857911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:05.857879 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw"] Oct 30 21:24:05.859912 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:24:05.859883 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5210006_c87e_440e_9a2f_a5db81390620.slice/crio-599a35c312aaa09dcf39feeb3d1d092d711c4259a460711b83084ed439cb24d8 WatchSource:0}: Error finding container 599a35c312aaa09dcf39feeb3d1d092d711c4259a460711b83084ed439cb24d8: Status 404 returned error can't find the container with id 599a35c312aaa09dcf39feeb3d1d092d711c4259a460711b83084ed439cb24d8 Oct 30 21:24:06.671972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:06.671936 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" event={"ID":"a5210006-c87e-440e-9a2f-a5db81390620","Type":"ContainerStarted","Data":"52a39de17e834fe8c17ddc9b2f0a53abcdce306114f30e63ff73b597d76a6391"} Oct 30 21:24:06.672422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:06.671982 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" event={"ID":"a5210006-c87e-440e-9a2f-a5db81390620","Type":"ContainerStarted","Data":"599a35c312aaa09dcf39feeb3d1d092d711c4259a460711b83084ed439cb24d8"} Oct 30 21:24:06.706077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:06.706029 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podStartSLOduration=1.7060170650000002 podStartE2EDuration="1.706017065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:06.704762671 +0000 UTC m=+885.914702300" watchObservedRunningTime="2026-10-30 21:24:06.706017065 +0000 UTC m=+885.915956694" Oct 30 21:24:07.292085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:07.292043 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:07.563860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:07.563760 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:08.888287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:08.888247 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:08.956807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:08.956774 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4"] Oct 30 21:24:08.960692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:08.960668 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:08.984259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:08.984227 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4"] Oct 30 21:24:09.095097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095052 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095105 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095221 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095263 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwhf7\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095370 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095499 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095564 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095594 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.095730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.095714 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.196878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.196795 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.196878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.196843 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.196936 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197003 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197064 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197114 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197170 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197196 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwhf7\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197327 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197842 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.197867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197859 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.198046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197977 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.198046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.197989 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.198445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.198418 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.199590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.199571 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.199667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.199631 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.214025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.214002 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.214414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.214374 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwhf7\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.274784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.274754 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:09.579002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.578929 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq"] Oct 30 21:24:09.583949 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:24:09.583920 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01841fb2_c4bb_4577_96b1_81b3756b0105.slice/crio-f17f82f62143b5d42c1b6f3caefe6002761a155052ec3c1bbb66c2e6fa2cecd5 WatchSource:0}: Error finding container f17f82f62143b5d42c1b6f3caefe6002761a155052ec3c1bbb66c2e6fa2cecd5: Status 404 returned error can't find the container with id f17f82f62143b5d42c1b6f3caefe6002761a155052ec3c1bbb66c2e6fa2cecd5 Oct 30 21:24:09.584743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.584724 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4"] Oct 30 21:24:09.584857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.584837 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.586497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.586464 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:09.586588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.586539 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:09.586636 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.586586 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:09.611405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.608336 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq"] Oct 30 21:24:09.693214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.693172 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" event={"ID":"01841fb2-c4bb-4577-96b1-81b3756b0105","Type":"ContainerStarted","Data":"298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119"} Oct 30 21:24:09.693414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.693239 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" event={"ID":"01841fb2-c4bb-4577-96b1-81b3756b0105","Type":"ContainerStarted","Data":"f17f82f62143b5d42c1b6f3caefe6002761a155052ec3c1bbb66c2e6fa2cecd5"} Oct 30 21:24:09.702101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702056 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702113 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702195 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702248 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702311 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702385 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702455 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjq4\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702499 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.702673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.702615 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.734205 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.734149 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podStartSLOduration=1.734132814 podStartE2EDuration="1.734132814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:09.730924988 +0000 UTC m=+888.940864620" watchObservedRunningTime="2026-10-30 21:24:09.734132814 +0000 UTC m=+888.944072444" Oct 30 21:24:09.803750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.803702 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.803954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.803766 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.803954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.803800 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.803954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.803863 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.803954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.803894 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.803966 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804033 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjq4\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804094 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804208 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804667 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804706 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804838 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.804959 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804908 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.805014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.804958 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.806641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.806614 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.806840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.806823 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.821073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.821046 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.821206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.821184 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjq4\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:09.900372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:09.900341 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:10.217305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.217281 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq"] Oct 30 21:24:10.219030 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:24:10.219003 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb133100_05d0_4614_b31a_6f9851985807.slice/crio-1bfd109203f79f08936e89e059b9cee8252d0aa6648909540574c79f77af9f7b WatchSource:0}: Error finding container 1bfd109203f79f08936e89e059b9cee8252d0aa6648909540574c79f77af9f7b: Status 404 returned error can't find the container with id 1bfd109203f79f08936e89e059b9cee8252d0aa6648909540574c79f77af9f7b Oct 30 21:24:10.221379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.221346 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:10.221493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.221429 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:10.221493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.221461 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:10.275837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.275812 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:10.277554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.277463 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:10.277554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.277510 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:10.535243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.535143 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:10.710025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.709970 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" event={"ID":"eb133100-05d0-4614-b31a-6f9851985807","Type":"ContainerStarted","Data":"6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8"} Oct 30 21:24:10.710217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.710066 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" event={"ID":"eb133100-05d0-4614-b31a-6f9851985807","Type":"ContainerStarted","Data":"1bfd109203f79f08936e89e059b9cee8252d0aa6648909540574c79f77af9f7b"} Oct 30 21:24:10.748155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.748098 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podStartSLOduration=1.748085499 podStartE2EDuration="1.748085499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.744919609 +0000 UTC m=+889.954859239" watchObservedRunningTime="2026-10-30 21:24:10.748085499 +0000 UTC m=+889.958025183" Oct 30 21:24:10.908429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:10.905494 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:11.296476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:11.286572 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:11.296476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:11.286648 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:11.899426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:11.899292 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:11.909447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:11.909051 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:24:11.909447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:11.909133 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:24:12.280471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:12.279765 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:12.280471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:12.279838 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:12.560481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:12.557650 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:12.903415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:12.901794 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:24:12.903415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:12.901860 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:24:13.278923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:13.278821 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:13.279562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:13.279525 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:13.907419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:13.906541 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:24:13.907419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:13.906615 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:24:14.278474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:14.277865 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:14.278474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:14.277936 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:14.724335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:14.724300 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:14.906417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:14.906247 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:24:14.906417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:14.906313 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:24:15.277265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.276797 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:15.277265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.276861 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:15.573471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.568541 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:15.617540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.617506 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:15.795381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.795351 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:24:15.901454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.901381 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:24:15.901627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:15.901485 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:24:16.275452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:16.275330 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:24:16.275452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:16.275413 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:24:16.767093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:16.767057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:17.198430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:17.194720 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:17.198430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:17.195552 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:17.198430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:17.196916 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 21:24:17.279114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:17.279087 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:17.279374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:17.279352 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:17.279960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:17.279943 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:24:18.473264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:18.473229 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:20.081516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:20.081465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:20.820664 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:20.820624 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:20.999358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:20.999325 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:24:21.003988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:21.003960 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:24:21.775443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:21.775410 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:23.383679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:23.383640 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:23.689920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:23.689876 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:25.007714 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.007677 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:25.031791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.031755 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7"] Oct 30 21:24:25.038037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.038007 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.052279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.052252 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b31518-openshift-default-dockercfg-xdgb8\"" Oct 30 21:24:25.064873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.064832 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7"] Oct 30 21:24:25.126883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.126843 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.126966 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127105 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127056 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127158 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127182 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127213 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127297 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127330 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxdv\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.127442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.127360 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.186212 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:25.186172 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:24:25.211772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.211745 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:24:25.228630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228601 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.228775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228635 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.228775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228660 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.228879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228783 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.228879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228824 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxdv\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.228879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228856 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228911 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228978 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.228987 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.229085 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.229247 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.229304 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.229416 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.229692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.229675 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.230982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.230963 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.231428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.231411 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.245202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.245171 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.245456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.245435 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxdv\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.358661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.358594 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:25.675019 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.674992 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7"] Oct 30 21:24:25.676544 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:24:25.676516 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48af0e61_5d81_4ab2_b651_121bd5fff894.slice/crio-6463506efcfe8839c35989259d487c0c3eae45141cad69ca9425018c42fd4d10 WatchSource:0}: Error finding container 6463506efcfe8839c35989259d487c0c3eae45141cad69ca9425018c42fd4d10: Status 404 returned error can't find the container with id 6463506efcfe8839c35989259d487c0c3eae45141cad69ca9425018c42fd4d10 Oct 30 21:24:25.678976 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.678942 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:25.679036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.679013 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:25.679083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.679045 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:25.855511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.855468 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" event={"ID":"48af0e61-5d81-4ab2-b651-121bd5fff894","Type":"ContainerStarted","Data":"4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff"} Oct 30 21:24:25.855688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.855537 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" event={"ID":"48af0e61-5d81-4ab2-b651-121bd5fff894","Type":"ContainerStarted","Data":"6463506efcfe8839c35989259d487c0c3eae45141cad69ca9425018c42fd4d10"} Oct 30 21:24:25.900143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:25.900081 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podStartSLOduration=0.900061777 podStartE2EDuration="900.061777ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:25.899277192 +0000 UTC m=+905.109216822" watchObservedRunningTime="2026-10-30 21:24:25.900061777 +0000 UTC m=+905.110001407" Oct 30 21:24:26.348182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:26.348141 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q"] Oct 30 21:24:26.370415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:26.359702 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:26.370415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:26.364452 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:24:26.370415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:26.364502 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:24:26.409329 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:26.407027 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:26.854969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:26.854911 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:27.380414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:27.364834 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:24:27.380414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:27.364915 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:24:28.369413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:28.360051 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:24:28.369413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:28.360113 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:24:28.488106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:28.488072 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:29.363865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:29.363814 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:24:29.364180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:29.364151 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:24:30.083891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:30.083857 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:30.361928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:30.361836 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:24:30.362172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:30.362140 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:24:31.373682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:31.373642 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:24:31.374850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:31.374807 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:24:31.787033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:31.786992 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:32.421141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:32.421100 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:32.421820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:32.421791 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:32.422782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:32.422763 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:24:33.507895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:33.507856 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:35.149935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:35.149903 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:36.788125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:36.788086 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:38.642095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:38.642061 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:40.690412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:40.690351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:41.541333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.541289 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" podUID="7bc5635e-40db-4b80-a5f0-58274c247fb8" containerName="main" containerID="cri-o://a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661" gracePeriod=45 Oct 30 21:24:41.814740 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.814710 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:24:41.867802 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.867766 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc\" (UniqueName: \"kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc\") pod \"7bc5635e-40db-4b80-a5f0-58274c247fb8\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " Oct 30 21:24:41.867968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.867889 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm\") pod \"7bc5635e-40db-4b80-a5f0-58274c247fb8\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " Oct 30 21:24:41.867968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.867937 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home\") pod \"7bc5635e-40db-4b80-a5f0-58274c247fb8\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " Oct 30 21:24:41.868095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.867973 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache\") pod \"7bc5635e-40db-4b80-a5f0-58274c247fb8\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " Oct 30 21:24:41.868095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868053 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir\") pod \"7bc5635e-40db-4b80-a5f0-58274c247fb8\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " Oct 30 21:24:41.868095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868076 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs\") pod \"7bc5635e-40db-4b80-a5f0-58274c247fb8\" (UID: \"7bc5635e-40db-4b80-a5f0-58274c247fb8\") " Oct 30 21:24:41.868297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868250 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home" pod "7bc5635e-40db-4b80-a5f0-58274c247fb8" (UID: "7bc5635e-40db-4b80-a5f0-58274c247fb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:41.868457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868321 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache" pod "7bc5635e-40db-4b80-a5f0-58274c247fb8" (UID: "7bc5635e-40db-4b80-a5f0-58274c247fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:41.868457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868422 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir" pod "7bc5635e-40db-4b80-a5f0-58274c247fb8" (UID: "7bc5635e-40db-4b80-a5f0-58274c247fb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:41.868633 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868615 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:24:41.868694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868637 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:24:41.868694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.868653 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:24:41.870111 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.870086 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc" pod "7bc5635e-40db-4b80-a5f0-58274c247fb8" (UID: "7bc5635e-40db-4b80-a5f0-58274c247fb8"). InnerVolumeSpecName "kube-api-access-44lkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:41.870254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.870236 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs" pod "7bc5635e-40db-4b80-a5f0-58274c247fb8" (UID: "7bc5635e-40db-4b80-a5f0-58274c247fb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:41.870340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.870321 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm" pod "7bc5635e-40db-4b80-a5f0-58274c247fb8" (UID: "7bc5635e-40db-4b80-a5f0-58274c247fb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:41.962056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.962020 5039 generic.go:356] "Generic (PLEG): container finished" podID="7bc5635e-40db-4b80-a5f0-58274c247fb8" containerID="a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661" exitCode=0 Oct 30 21:24:41.962202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.962063 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" Oct 30 21:24:41.962202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.962093 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" event={"ID":"7bc5635e-40db-4b80-a5f0-58274c247fb8","Type":"ContainerDied","Data":"a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661"} Oct 30 21:24:41.962202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.962121 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q" event={"ID":"7bc5635e-40db-4b80-a5f0-58274c247fb8","Type":"ContainerDied","Data":"7898ae1d099b669b1b64db3151263d2f8690975d02c09b357b6435abc5a501b2"} Oct 30 21:24:41.962202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.962137 5039 scope.go:122] "RemoveContainer" containerID="a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661" Oct 30 21:24:41.969879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.969841 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc5635e-40db-4b80-a5f0-58274c247fb8-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:24:41.969879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.969862 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lkc\" (UniqueName: \"kubernetes.io/projected/7bc5635e-40db-4b80-a5f0-58274c247fb8-kube-api-access-44lkc\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:24:41.969879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.969877 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc5635e-40db-4b80-a5f0-58274c247fb8-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:24:41.970067 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:41.969940 5039 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:41.970067 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:41.969957 5039 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:41.970067 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:41.970064 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials podName:ca512b1f-81ec-4f41-9178-d78794fd01c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:42.470045441 +0000 UTC m=+921.679985065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials") pod "payload-processing-external-model-5c46ddcfbd-8xjq8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8") : secret "agc-cred-9d93cf183177" not found Oct 30 21:24:41.972849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.972832 5039 scope.go:122] "RemoveContainer" containerID="a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661" Oct 30 21:24:41.973117 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:41.973099 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661\": container with ID starting with a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661 not found: ID does not exist" containerID="a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661" Oct 30 21:24:41.973158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:41.973125 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661"} err="failed to get container status \"a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661\": rpc error: code = NotFound desc = could not find container \"a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661\": container with ID starting with a9c5e04ddae25a164c30bb0b681cdce2fedf8627402952baa7593873d17d7661 not found: ID does not exist" Oct 30 21:24:42.006597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:42.006569 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q"] Oct 30 21:24:42.010312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:42.010288 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-a-26510fbf-kserve-7778769fc9-dqd9q"] Oct 30 21:24:42.408073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:42.408036 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:42.476849 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:42.476814 5039 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:42.476849 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:42.476840 5039 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:42.477053 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:42.476902 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials podName:ca512b1f-81ec-4f41-9178-d78794fd01c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:43.47688887 +0000 UTC m=+922.686828479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials") pod "payload-processing-external-model-5c46ddcfbd-8xjq8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8") : secret "agc-cred-9d93cf183177" not found Oct 30 21:24:42.921738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:42.921696 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc5635e-40db-4b80-a5f0-58274c247fb8" path="/var/lib/kubelet/pods/7bc5635e-40db-4b80-a5f0-58274c247fb8/volumes" Oct 30 21:24:43.489119 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:43.489089 5039 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:43.489119 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:43.489113 5039 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:43.489328 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:43.489174 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials podName:ca512b1f-81ec-4f41-9178-d78794fd01c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:45.489159846 +0000 UTC m=+924.699099459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials") pod "payload-processing-external-model-5c46ddcfbd-8xjq8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8") : secret "agc-cred-9d93cf183177" not found Oct 30 21:24:44.027701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:44.027659 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:44.169207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:44.169158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:45.020190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.020162 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:24:45.510669 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:45.510638 5039 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:45.510669 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:45.510667 5039 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8: secret "agc-cred-9d93cf183177" not found Oct 30 21:24:45.511079 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:45.510733 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials podName:ca512b1f-81ec-4f41-9178-d78794fd01c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:49.51071403 +0000 UTC m=+928.720653641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials") pod "payload-processing-external-model-5c46ddcfbd-8xjq8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8") : secret "agc-cred-9d93cf183177" not found Oct 30 21:24:45.790048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.789958 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:45.979000 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.978967 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b"] Oct 30 21:24:45.979493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.979479 5039 cpu_manager.go:414] "containerMap: removing container" podUID="7bc5635e-40db-4b80-a5f0-58274c247fb8" containerName="main" Oct 30 21:24:45.979493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.979494 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc5635e-40db-4b80-a5f0-58274c247fb8" containerName="main" Oct 30 21:24:45.979610 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.979584 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc5635e-40db-4b80-a5f0-58274c247fb8" containerName="main" Oct 30 21:24:45.984281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:45.984260 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.014589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.014554 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"distinct-2-w1-b31518-kserve-self-signed-certs\"" Oct 30 21:24:46.015816 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.015785 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b"] Oct 30 21:24:46.117086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.116989 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:46.117266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.117160 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.117266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.117212 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.117410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.117278 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.117410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.117330 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.117410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.117362 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.117619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.117594 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cpvs\" (UniqueName: \"kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.218596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.218551 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.218808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.218721 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cpvs\" (UniqueName: \"kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.218883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.218814 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.218883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.218870 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.218988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.218903 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.218988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.218937 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.219587 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.219552 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.219793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.219684 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.220213 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.220192 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.223048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.223022 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.223290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.223271 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.240873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.240845 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cpvs\" (UniqueName: \"kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs\") pod \"distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.298724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.298682 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:46.611017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.610981 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b"] Oct 30 21:24:46.614958 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:24:46.614915 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711dd07d_b60e_4a12_82a6_da528fb1d22b.slice/crio-f6fab2e56c8837b2314954f6bfe2f434d90fdcbc43a2724068365db5a71e2324 WatchSource:0}: Error finding container f6fab2e56c8837b2314954f6bfe2f434d90fdcbc43a2724068365db5a71e2324: Status 404 returned error can't find the container with id f6fab2e56c8837b2314954f6bfe2f434d90fdcbc43a2724068365db5a71e2324 Oct 30 21:24:46.995451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.995378 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" event={"ID":"711dd07d-b60e-4a12-82a6-da528fb1d22b","Type":"ContainerStarted","Data":"757ffcf1766cdf5e17eedde29858efacc91b82399de87c63b68a1a0009a5133f"} Oct 30 21:24:46.995651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:46.995468 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" event={"ID":"711dd07d-b60e-4a12-82a6-da528fb1d22b","Type":"ContainerStarted","Data":"f6fab2e56c8837b2314954f6bfe2f434d90fdcbc43a2724068365db5a71e2324"} Oct 30 21:24:47.032871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:47.032823 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podStartSLOduration=2.032810573 podStartE2EDuration="2.032810573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:47.030676003 +0000 UTC m=+926.240615635" watchObservedRunningTime="2026-10-30 21:24:47.032810573 +0000 UTC m=+926.242750202" Oct 30 21:24:47.298846 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:47.298767 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:47.600242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:47.600132 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:49.535745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:49.535709 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:49.566791 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:49.566764 5039 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-18092bbdb3eb: secret "agc-cred-18092bbdb3eb" not found Oct 30 21:24:49.566791 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:49.566791 5039 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8: secret "agc-cred-18092bbdb3eb" not found Oct 30 21:24:49.567025 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:49.566850 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials podName:ca512b1f-81ec-4f41-9178-d78794fd01c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:57.566836084 +0000 UTC m=+936.776775704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials") pod "payload-processing-external-model-5c46ddcfbd-8xjq8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8") : secret "agc-cred-18092bbdb3eb" not found Oct 30 21:24:50.905125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:50.905060 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:53.214974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:53.214932 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:54.153253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.153218 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr"] Oct 30 21:24:54.157481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.157460 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.201329 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.201297 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr"] Oct 30 21:24:54.227794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.227762 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:54.324158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324123 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324194 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324294 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324331 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97swq\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324464 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324497 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324532 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324588 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.324657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.324644 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426149 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426196 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426233 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426262 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426333 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426433 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426499 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426557 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426593 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97swq\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426727 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426828 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.426961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426948 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.427032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.426991 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.427168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.427144 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.428509 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.428492 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.428897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.428877 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.444716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.444693 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.448354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.448334 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97swq\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.474078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.474053 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:54.721021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.720929 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5"] Oct 30 21:24:54.726551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.726519 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.755468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.752615 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5"] Oct 30 21:24:54.800176 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.800134 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr"] Oct 30 21:24:54.829043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.828996 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:54.829187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.829091 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:54.829187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.829128 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:54.833187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833153 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833243 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833291 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntd9c\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833323 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833476 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833530 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833567 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833591 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.833652 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.833620 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.934808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.934773 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.934833 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntd9c\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.934863 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.934945 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.934990 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.935034 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.935075 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.935100 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.935145 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.935477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.935321 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.936138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.936082 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.936487 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.936424 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.936791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.936768 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.937717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.937695 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.939041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.939020 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.941609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.941590 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.953115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.952727 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:54.953115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:54.953006 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntd9c\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:55.044440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.044336 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" event={"ID":"9f69bd36-dade-487d-bba3-db4ce7669a44","Type":"ContainerStarted","Data":"644f03a57234dd825bf9e1414615affd2c446c5e2eb336e0d693b2705faa92bf"} Oct 30 21:24:55.044640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.044617 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" event={"ID":"9f69bd36-dade-487d-bba3-db4ce7669a44","Type":"ContainerStarted","Data":"f062a329af53ebfc32e950e6a78f871b056ffa0ef34e4af2fb5a8cc7f535c5c3"} Oct 30 21:24:55.063601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.063569 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:55.083947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.083886 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podStartSLOduration=1.0838672 podStartE2EDuration="1.0838672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:55.081039037 +0000 UTC m=+934.290978666" watchObservedRunningTime="2026-10-30 21:24:55.0838672 +0000 UTC m=+934.293806831" Oct 30 21:24:55.298468 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:55.297186 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:24:55.464696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.464265 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5"] Oct 30 21:24:55.473244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.472638 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:55.473244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.472729 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:55.473244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.472775 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:55.507426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.500453 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:24:55.520222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.519852 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:24:55.520222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:55.519908 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:24:56.105847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.105814 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" event={"ID":"47e7b98a-0189-4670-901b-1fa3753a86fc","Type":"ContainerStarted","Data":"a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b"} Oct 30 21:24:56.106101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.106077 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" event={"ID":"47e7b98a-0189-4670-901b-1fa3753a86fc","Type":"ContainerStarted","Data":"7d6287c6c1ee97d021f7c89ca42259363f5e75dc75cc72abad0f58a491c09678"} Oct 30 21:24:56.170255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.170180 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podStartSLOduration=2.170158813 podStartE2EDuration="2.170158813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:56.16706423 +0000 UTC m=+935.377003861" watchObservedRunningTime="2026-10-30 21:24:56.170158813 +0000 UTC m=+935.380098446" Oct 30 21:24:56.304953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.303595 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:56.478901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.478801 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:24:56.479173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.479137 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:24:56.567549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.567500 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:56.613532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.613477 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:56.684551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.684505 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:56.721538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.721489 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:56.822649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:56.822524 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:57.066069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.066038 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:24:57.066431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.066403 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:24:57.066606 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.066583 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:24:57.178451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.174710 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:57.206988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.206951 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:57.482892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.482794 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:24:57.483516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.483482 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:24:57.568067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.567999 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:57.568831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.568805 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:24:57.647542 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:57.647510 5039 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-18092bbdb3eb: secret "agc-cred-18092bbdb3eb" not found Oct 30 21:24:57.647774 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:57.647756 5039 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8: secret "agc-cred-18092bbdb3eb" not found Oct 30 21:24:57.647939 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:24:57.647928 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials podName:ca512b1f-81ec-4f41-9178-d78794fd01c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:25:13.647906394 +0000 UTC m=+952.857846016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials") pod "payload-processing-external-model-5c46ddcfbd-8xjq8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8") : secret "agc-cred-18092bbdb3eb" not found Oct 30 21:24:57.685561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.685512 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:57.686494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.686467 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:24:57.760771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.760675 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:57.805460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.798963 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:24:57.913454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.857525 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:57.957456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.936404 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:57.957456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:57.937150 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:24:58.103867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.061522 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:58.104496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.104416 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:24:58.104725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.104470 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:24:58.178743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.178699 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:58.477918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.477885 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:24:58.478232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.478185 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:24:58.580896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.580848 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:24:58.717414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.688556 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:58.787055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.786954 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 21:24:58.871555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.869829 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:58.871555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.870571 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:24:58.954554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:58.954512 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:59.087833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.087661 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:24:59.091456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.088073 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:24:59.179151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.179102 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:59.525120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.525086 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:24:59.525429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.525376 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:24:59.623411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.622724 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:59.692587 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.692526 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:59.796708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.796599 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:59.807615 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.803287 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:24:59.876904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.876787 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:59.955797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.955768 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:24:59.958567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:24:59.957079 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:00.065102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.065011 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:25:00.065341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.065301 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:25:00.140764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.126687 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:25:00.296414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.285768 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:25:00.490413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.471550 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8"] Oct 30 21:25:00.695750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.478496 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:25:00.695750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.694096 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:25:00.717072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.717028 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:25:00.746667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.746443 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:00.841553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.841507 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:00.901416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.886474 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:00.952136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:00.952103 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:25:01.050254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:01.050174 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:25:01.073422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:01.072797 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:25:01.073422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:01.072851 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:25:01.137513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:01.136455 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:25:01.483415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:01.477864 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:25:01.483415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:01.477940 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:25:02.085540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:02.085496 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:25:02.086137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:02.086098 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:25:02.500877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:02.500840 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:25:02.501222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:02.501189 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:25:02.830434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:02.830310 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:03.066536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:03.066499 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:25:03.066803 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:03.066770 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:25:03.476852 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:03.476813 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:25:03.477476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:03.477438 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:25:04.064414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:04.064293 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:25:04.064414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:04.064354 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:25:04.479069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:04.479030 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:25:04.479526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:04.479505 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:25:04.480108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:04.480089 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:25:04.615301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:04.615264 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:05.372169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.372130 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:25:05.373029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.372998 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:25:05.374797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.374769 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:25:05.695145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.695104 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" podUID="ca512b1f-81ec-4f41-9178-d78794fd01c8" containerName="payload-processing" containerID="cri-o://118d314a39abd6dc0a4ff2d64827dedef97ef1a4f726a0883f58b49c3e008eda" gracePeriod=25 Oct 30 21:25:05.855071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.855037 5039 generic.go:356] "Generic (PLEG): container finished" podID="ca512b1f-81ec-4f41-9178-d78794fd01c8" containerID="118d314a39abd6dc0a4ff2d64827dedef97ef1a4f726a0883f58b49c3e008eda" exitCode=0 Oct 30 21:25:05.855248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.855109 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" event={"ID":"ca512b1f-81ec-4f41-9178-d78794fd01c8","Type":"ContainerDied","Data":"118d314a39abd6dc0a4ff2d64827dedef97ef1a4f726a0883f58b49c3e008eda"} Oct 30 21:25:05.974109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:05.974086 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:25:06.069092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069059 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume\") pod \"ca512b1f-81ec-4f41-9178-d78794fd01c8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " Oct 30 21:25:06.069280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069113 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp\") pod \"ca512b1f-81ec-4f41-9178-d78794fd01c8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " Oct 30 21:25:06.069280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069153 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials\") pod \"ca512b1f-81ec-4f41-9178-d78794fd01c8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " Oct 30 21:25:06.069280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069204 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume\") pod \"ca512b1f-81ec-4f41-9178-d78794fd01c8\" (UID: \"ca512b1f-81ec-4f41-9178-d78794fd01c8\") " Oct 30 21:25:06.069589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069556 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume" pod "ca512b1f-81ec-4f41-9178-d78794fd01c8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:06.069707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069678 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume" pod "ca512b1f-81ec-4f41-9178-d78794fd01c8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:06.069707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.069689 5039 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-routing-overlay-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:25:06.071437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.071414 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials" pod "ca512b1f-81ec-4f41-9178-d78794fd01c8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:06.071543 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.071465 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp" pod "ca512b1f-81ec-4f41-9178-d78794fd01c8" (UID: "ca512b1f-81ec-4f41-9178-d78794fd01c8"). InnerVolumeSpecName "kube-api-access-g8tlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:06.171414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.171363 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tlp\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-kube-api-access-g8tlp\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:25:06.171550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.171422 5039 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca512b1f-81ec-4f41-9178-d78794fd01c8-provider-credentials\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:25:06.171550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.171433 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca512b1f-81ec-4f41-9178-d78794fd01c8-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:25:06.284105 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.284015 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:06.861973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.861940 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" Oct 30 21:25:06.862382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.861974 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8" event={"ID":"ca512b1f-81ec-4f41-9178-d78794fd01c8","Type":"ContainerDied","Data":"cf6f07f10c7134a8b7c623f39c32e47a4802c03c165d7d317c0151d0ee0e52bb"} Oct 30 21:25:06.862382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.862036 5039 scope.go:122] "RemoveContainer" containerID="118d314a39abd6dc0a4ff2d64827dedef97ef1a4f726a0883f58b49c3e008eda" Oct 30 21:25:06.906947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.906919 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8"] Oct 30 21:25:06.910467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.910446 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c46ddcfbd-8xjq8"] Oct 30 21:25:06.921295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:06.921271 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca512b1f-81ec-4f41-9178-d78794fd01c8" path="/var/lib/kubelet/pods/ca512b1f-81ec-4f41-9178-d78794fd01c8/volumes" Oct 30 21:25:08.052650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:08.052611 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:09.090262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:09.090223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:09.230430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:09.230373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:09.846889 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:09.846850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:10.903715 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:10.903684 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:25:11.596449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:11.596414 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:12.009423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:12.009353 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:13.832716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:13.832677 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:15.392665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:15.392623 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:16.999798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:16.999760 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:17.533676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.533635 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg"] Oct 30 21:25:17.535124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.535090 5039 cpu_manager.go:414] "containerMap: removing container" podUID="ca512b1f-81ec-4f41-9178-d78794fd01c8" containerName="payload-processing" Oct 30 21:25:17.535124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.535124 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca512b1f-81ec-4f41-9178-d78794fd01c8" containerName="payload-processing" Oct 30 21:25:17.535432 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.535414 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca512b1f-81ec-4f41-9178-d78794fd01c8" containerName="payload-processing" Oct 30 21:25:17.546563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.546535 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.553920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.553893 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg"] Oct 30 21:25:17.559405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.559361 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"kube-root-ca.crt\"" Oct 30 21:25:17.559582 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.559546 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"distinct-2-w0-49738b-kserve-self-signed-certs\"" Oct 30 21:25:17.568760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.568728 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"default-dockercfg-8qqf5\"" Oct 30 21:25:17.568760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.568727 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"openshift-service-ca.crt\"" Oct 30 21:25:17.642561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.642526 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.642715 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.642691 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.642762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.642729 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.642762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.642753 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzmn\" (UniqueName: \"kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.642905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.642886 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.643006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.642992 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.650598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.650569 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:17.744439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744400 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.744644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744514 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.744708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744649 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.744708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744687 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.744818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744723 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzmn\" (UniqueName: \"kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.744818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744770 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.744818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.744802 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.745222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.745197 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.745481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.745456 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.748074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.747950 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.774723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.771037 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.775781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.775754 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzmn\" (UniqueName: \"kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn\") pod \"distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:17.863165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:17.863072 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:18.164251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:18.164222 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg"] Oct 30 21:25:18.165485 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:25:18.165460 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66188eb2_2660_4cca_9fcf_e70428ee5286.slice/crio-454424fd6c86a8994fd904acd346d2ef0d7217aaee4371f64310640a5d2ee792 WatchSource:0}: Error finding container 454424fd6c86a8994fd904acd346d2ef0d7217aaee4371f64310640a5d2ee792: Status 404 returned error can't find the container with id 454424fd6c86a8994fd904acd346d2ef0d7217aaee4371f64310640a5d2ee792 Oct 30 21:25:18.783264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:18.783217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:18.956124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:18.956083 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" event={"ID":"66188eb2-2660-4cca-9fcf-e70428ee5286","Type":"ContainerStarted","Data":"48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220"} Oct 30 21:25:18.956293 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:18.956130 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" event={"ID":"66188eb2-2660-4cca-9fcf-e70428ee5286","Type":"ContainerStarted","Data":"454424fd6c86a8994fd904acd346d2ef0d7217aaee4371f64310640a5d2ee792"} Oct 30 21:25:18.992969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:18.992909 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podStartSLOduration=1.9928919920000001 podStartE2EDuration="1.992891992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:18.99043944 +0000 UTC m=+958.200379072" watchObservedRunningTime="2026-10-30 21:25:18.992891992 +0000 UTC m=+958.202831624" Oct 30 21:25:19.863934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:19.863899 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:20.412807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:20.412770 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:21.979987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:21.979951 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:23.586693 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:23.586656 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:25.187697 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:25:25.187650 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:25:25.217644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.217611 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm"] Oct 30 21:25:25.223149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.223120 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.255355 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.255323 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm"] Oct 30 21:25:25.335637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335600 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskqh\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.335820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335745 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.335820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335808 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.335906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335847 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.335959 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335946 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.336004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335967 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.336004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.335987 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.336004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.336001 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.336098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.336047 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.428075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.428037 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:25.436925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.436899 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.436936 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.436956 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.436979 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437013 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437039 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437062 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437152 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskqh\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437273 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437424 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437434 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437524 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437632 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.437888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.437858 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.439386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.439365 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.439487 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.439457 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.455118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.455095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.455214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.455199 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskqh\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:25.548872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:25.548577 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:26.008130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:26.008031 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm"] Oct 30 21:25:26.022424 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:25:26.018367 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd416065e_0217_4c68_b041_cdb3383d55e7.slice/crio-892a03aa34658c8b6eb586b4b644dd5a2ce37a805654b33d1a9f4483b83b5553 WatchSource:0}: Error finding container 892a03aa34658c8b6eb586b4b644dd5a2ce37a805654b33d1a9f4483b83b5553: Status 404 returned error can't find the container with id 892a03aa34658c8b6eb586b4b644dd5a2ce37a805654b33d1a9f4483b83b5553 Oct 30 21:25:26.033952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:26.033899 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:26.034099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:26.034004 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:26.034099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:26.034050 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:27.058437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.057727 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" event={"ID":"d416065e-0217-4c68-b041-cdb3383d55e7","Type":"ContainerStarted","Data":"1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89"} Oct 30 21:25:27.058437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.057813 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" event={"ID":"d416065e-0217-4c68-b041-cdb3383d55e7","Type":"ContainerStarted","Data":"892a03aa34658c8b6eb586b4b644dd5a2ce37a805654b33d1a9f4483b83b5553"} Oct 30 21:25:27.517070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.517033 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:27.553543 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.553510 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:27.553884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.553862 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:27.554021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.553998 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:27.869471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:27.868538 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:28.523827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.523786 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:28.544810 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.544778 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:28.559873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.559829 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:28.561456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.561420 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:28.562024 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.560646 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podStartSLOduration=3.560626602 podStartE2EDuration="3.560626602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:27.135057079 +0000 UTC m=+966.344996708" watchObservedRunningTime="2026-10-30 21:25:28.560626602 +0000 UTC m=+967.770566232" Oct 30 21:25:28.803583 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.803479 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:28.975519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:28.975475 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:29.550564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:29.550529 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:29.550944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:29.550582 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:30.552244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:30.550912 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:30.552244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:30.550970 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:30.690446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:30.690406 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:31.550401 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:31.550355 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:31.550591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:31.550432 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:31.993942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:31.993896 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:32.550460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:32.550426 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:32.550632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:32.550483 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:33.108750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:33.108712 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:33.550738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:33.550696 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:25:33.550943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:33.550752 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 21:25:34.555033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:34.555003 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:34.555450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:34.555430 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:34.556063 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:34.556046 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 21:25:35.107525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:35.107493 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:36.984032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:36.983993 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:38.672658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:38.672620 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:40.236532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.236484 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:40.243549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.243517 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n"] Oct 30 21:25:40.248386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.248361 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.276453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.276378 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n"] Oct 30 21:25:40.319520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319482 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.319734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319535 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.319734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319640 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.319734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319715 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.319917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319816 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.319917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319899 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.320021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.319941 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.320174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.320153 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.320227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.320211 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bhg\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.421938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.421896 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.421952 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.421990 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422095 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422144 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bhg\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422229 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422258 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422291 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422344 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422347 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422485 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422710 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422650 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422770 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.422912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.422891 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.424507 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.424476 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.424750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.424733 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.438694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.438672 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.438784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.438752 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bhg\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.565928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.565859 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:40.877022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.876993 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n"] Oct 30 21:25:40.878265 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:25:40.878240 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdabfed_f775_408b_9ab9_b9e5cc1e653f.slice/crio-3abd468c37043a516beaaeca697d105bd797dd04f3921ee42ffa993f2a6b017c WatchSource:0}: Error finding container 3abd468c37043a516beaaeca697d105bd797dd04f3921ee42ffa993f2a6b017c: Status 404 returned error can't find the container with id 3abd468c37043a516beaaeca697d105bd797dd04f3921ee42ffa993f2a6b017c Oct 30 21:25:40.880996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.880961 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:40.881081 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.881039 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:40.881120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:40.881079 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:41.353353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.353310 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" event={"ID":"bfdabfed-f775-408b-9ab9-b9e5cc1e653f","Type":"ContainerStarted","Data":"10fe92eb3e4aa8dddc73531631a830863433936289f3fa7f37d9a780494bcf48"} Oct 30 21:25:41.353772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.353373 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" event={"ID":"bfdabfed-f775-408b-9ab9-b9e5cc1e653f","Type":"ContainerStarted","Data":"3abd468c37043a516beaaeca697d105bd797dd04f3921ee42ffa993f2a6b017c"} Oct 30 21:25:41.400862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.400813 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podStartSLOduration=1.40079913 podStartE2EDuration="1.40079913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:41.397815241 +0000 UTC m=+980.607754870" watchObservedRunningTime="2026-10-30 21:25:41.40079913 +0000 UTC m=+980.610738838" Oct 30 21:25:41.566122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.566091 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:41.567662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.567637 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:41.567788 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.567689 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:41.770855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:41.770810 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:42.575416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:42.572834 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:42.575416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:42.572892 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:42.752436 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:42.752379 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:25:43.512416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:43.511927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:43.568907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:43.568872 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:43.569227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:43.569184 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:44.567092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.567054 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:44.568067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.567702 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:44.783416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.782987 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:44.809413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.803462 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:44.892207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.891470 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:44.958595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.958548 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:44.972540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:44.972474 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:45.193739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.193638 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:45.537814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.537703 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:45.568566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.568517 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:45.569199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.569150 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:45.591167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.591125 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:45.692552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.691477 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 21:25:45.788668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.788539 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:45.789903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.789882 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:25:45.804428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.804367 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:45.805068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.805045 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:25:45.992641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:45.992593 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:46.043614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.040499 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:25:46.045204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.044328 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:25:46.588457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.577511 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:46.636440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.635381 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:46.636643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.636601 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:46.741589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.741549 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:46.741897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.741867 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:46.858734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:46.858626 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:47.045414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.044865 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 21:25:47.286518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.286471 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:47.360671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.360622 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:47.447276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.447231 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:47.600627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.600531 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:47.601207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.601159 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:47.741696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.741632 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:47.859710 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.859598 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:47.925561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.925512 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hcx4p" podUID="7732615f-2c18-4474-8149-8a6798b4f6fe" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.41:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:47.994708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:47.994662 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.052672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.052569 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.052672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.052651 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 21:25:48.054972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.053016 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 21:25:48.207411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.206691 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:25:48.207411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.206763 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.385417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.357731 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:25:48.385729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.385690 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.422416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.412327 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:25:48.422416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.412717 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:25:48.455572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.455523 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:25:48.455912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.455876 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.488016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.487532 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.489560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.489510 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:25:48.489821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.489771 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:48.744954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.744852 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 21:25:48.832424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.774066 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:48.891328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.774105 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.048424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.889098 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.048650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.890327 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:25:49.049194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:48.999182 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.049331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.046069 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:25:49.049458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.049439 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.049565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.049116 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:49.070419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.058019 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.070419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.058118 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 21:25:49.070419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.065781 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:25:49.070419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.067551 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:25:49.070419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.067602 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:25:49.156628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.156588 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:25:49.156911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.156878 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.469421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.439216 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.686438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.650455 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:49.686438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.678704 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:49.687233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.687202 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:49.687456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:49.687428 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:50.062414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.054353 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:50.062414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.054567 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:25:50.062953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.062474 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:50.091562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.091516 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:50.091929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.091899 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:25:50.093875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.093847 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:50.300425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.285492 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:50.463415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.459069 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:25:50.486539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.485723 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:25:50.519440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.516608 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:25:50.557324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.541255 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:25:50.557324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.545058 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:25:50.557324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.555503 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:25:50.581602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.581556 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:50.581885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.581840 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:50.621523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.621493 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:25:50.628350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.628323 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:25:50.637140 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:50.637115 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:25:51.208159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:51.208112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:51.567055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:51.566967 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:25:51.567055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:51.567026 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:25:53.438882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:53.438844 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:53.439378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:53.439247 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:53.440285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:53.440262 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:25:53.481367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:53.481325 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:55.186939 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:25:55.186898 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:25:55.203874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:55.203842 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:56.994867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:56.994832 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:58.704277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:25:58.704230 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:00.343213 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:00.343164 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:02.103050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:02.103010 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:04.639410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:04.638514 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:04.655415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:04.653539 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:05.312616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:05.312576 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:07.192072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:07.192034 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:08.895911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:08.895869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:10.506434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:10.506382 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:12.115795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:12.115759 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:12.252175 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:12.252138 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:13.112598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.112567 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz"] Oct 30 21:26:13.116850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.116829 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.141654 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.141628 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz"] Oct 30 21:26:13.203718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.203680 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.203906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.203741 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.203906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.203849 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.204036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.203944 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.204036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.204008 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftb2\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.204036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.204030 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.204209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.204047 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.204272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.204203 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.204328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.204283 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305355 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305320 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305433 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftb2\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305472 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305488 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305517 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305563 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305663 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305716 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305774 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.305971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.305923 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.306093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.306065 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.306214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.306118 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.306214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.306096 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.306651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.306627 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.307899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.307878 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.308258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.308235 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.321890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.321860 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.322117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.322099 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftb2\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.430946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.430918 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:13.742587 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.742560 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz"] Oct 30 21:26:13.743936 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:26:13.743907 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57417a1b_d284_4aac_9a1d_f7843068d67e.slice/crio-121c3c0c05755c628775a1330cc4aadbcfd8982fd0e70ce929ef1114d43ed775 WatchSource:0}: Error finding container 121c3c0c05755c628775a1330cc4aadbcfd8982fd0e70ce929ef1114d43ed775: Status 404 returned error can't find the container with id 121c3c0c05755c628775a1330cc4aadbcfd8982fd0e70ce929ef1114d43ed775 Oct 30 21:26:13.747668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.746789 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:26:13.747668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.746881 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:26:13.747668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:13.746944 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:26:14.000370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.000285 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:14.391546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.391513 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" event={"ID":"57417a1b-d284-4aac-9a1d-f7843068d67e","Type":"ContainerStarted","Data":"62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab"} Oct 30 21:26:14.391951 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.391568 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" event={"ID":"57417a1b-d284-4aac-9a1d-f7843068d67e","Type":"ContainerStarted","Data":"121c3c0c05755c628775a1330cc4aadbcfd8982fd0e70ce929ef1114d43ed775"} Oct 30 21:26:14.431838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.431804 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:14.433625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.433597 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:14.433747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.433645 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:14.449984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.449928 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podStartSLOduration=1.449908883 podStartE2EDuration="1.449908883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:14.436780562 +0000 UTC m=+1013.646720191" watchObservedRunningTime="2026-10-30 21:26:14.449908883 +0000 UTC m=+1013.659848516" Oct 30 21:26:14.835130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:14.835045 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:26:15.432548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:15.432513 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:15.433199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:15.433141 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:15.685470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:15.684431 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:16.433074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:16.433035 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:16.433731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:16.433698 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:17.308623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:17.308551 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:17.432902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:17.432867 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:17.433196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:17.433164 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:17.650417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:17.642496 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 21:26:18.114470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:18.108498 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:18.546563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:18.546515 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:18.564562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:18.564513 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:18.609207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:18.609166 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:18.609442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:18.609231 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:19.432414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:19.432348 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:19.432593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:19.432421 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:19.783111 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:19.783014 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:20.431464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:20.431432 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:20.433706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:20.431487 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:21.431945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:21.431912 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:26:21.432326 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:21.431968 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:26:21.727961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:21.727856 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:23.119752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:23.119723 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:23.120276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:23.120126 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:23.121058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:23.121032 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 21:26:23.479988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:23.479943 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:24.578687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:24.578639 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:24.717845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:24.717806 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:25.188278 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:26:25.188238 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:26:25.314097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:25.314059 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:25.458948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:25.458834 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:25.624461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:25.623990 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:25.780816 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:25.780724 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:25.950135 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:25.950100 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:26.877898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:26.877856 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.025962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:27.025924 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.236157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:27.236128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.407537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:27.407497 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:29.093722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:29.093682 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:30.697708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:30.697669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:33.017096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.016835 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:33.540416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.536800 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:33.560434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.559985 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:33.562454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.562147 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:26:33.658533 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.658485 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:33.996425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.993565 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:33.996425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:33.993669 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:34.073622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.073554 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:34.097536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.097474 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:34.121861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.121821 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:34.122251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.122123 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:34.584086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.584047 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:26:34.584638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.584620 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:26:34.601737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.590791 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:34.606423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.602223 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:26:34.669050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.663044 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 21:26:34.669050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.663480 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:26:34.769416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.741063 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:26:34.825343 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.824310 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:26:34.872574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:34.872471 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:35.022411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.018046 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:35.096415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.093410 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:35.124469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.113859 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 21:26:35.204434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.168153 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:26:35.204434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.168199 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q"] Oct 30 21:26:35.234481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.216181 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:35.234995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.234970 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:26:35.247414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.235991 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.266621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.266533 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q"] Oct 30 21:26:35.313443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.276523 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"distinct-w2-9d677d-kserve-self-signed-certs\"" Oct 30 21:26:35.316786 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.316754 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.317065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.317042 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfhn\" (UniqueName: \"kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.317445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.317422 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.317717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.317690 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.437110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.340864 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.437110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.340906 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.447838 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.447894 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.447983 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.448028 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfhn\" (UniqueName: \"kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.448139 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.448214 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.448777 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.449252 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.484454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.468738 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.554774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.554154 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:26:35.666787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.666735 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfhn\" (UniqueName: \"kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.667309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.667280 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:35.932423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.931771 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:35.955416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:35.949737 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:26:36.188189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.187914 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:36.200441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.197716 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs\") pod \"distinct-w2-9d677d-kserve-77cd49777d-mgd2q\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:36.208455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.206655 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:36.254286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.254227 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 21:26:36.355422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.353225 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:36.461469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.460815 5039 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:36.461469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.460867 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:36.577932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.577892 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 21:26:36.669414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.654527 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:36.894425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.869490 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:36.941883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:36.941488 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:26:37.151494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.113463 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:37.168853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.168820 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:26:37.169602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.169584 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:26:37.169773 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.169761 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:26:37.297974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.294694 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q"] Oct 30 21:26:37.569488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.568340 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" event={"ID":"212834dc-cb28-4598-a82f-1d1b4b18d0c9","Type":"ContainerStarted","Data":"4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2"} Oct 30 21:26:37.569488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.568430 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" event={"ID":"212834dc-cb28-4598-a82f-1d1b4b18d0c9","Type":"ContainerStarted","Data":"6c2d3820b88a939b161b59e94ec4e56dce04928e120594fc3fe0a142e78f41b1"} Oct 30 21:26:37.604866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:37.604801 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podStartSLOduration=3.6047829609999997 podStartE2EDuration="3.604782961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:37.602011316 +0000 UTC m=+1036.811950945" watchObservedRunningTime="2026-10-30 21:26:37.604782961 +0000 UTC m=+1036.814722593" Oct 30 21:26:38.368835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:38.368800 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:39.492305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:39.492270 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:39.667691 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:39.667653 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:41.516323 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:41.516284 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:43.084266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.084234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:43.223601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.223571 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml"] Oct 30 21:26:43.227288 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.227270 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.248479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.248455 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml"] Oct 30 21:26:43.263498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263464 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.263646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263630 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.263711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263667 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.263711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263687 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.263819 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263763 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.264157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263824 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.264157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263851 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.264157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263880 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.264157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.263900 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77w88\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.364984 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365051 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365082 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365125 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365179 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365208 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365251 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365281 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77w88\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365363 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365694 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.365958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365792 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.366265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365975 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.366265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.365976 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.366265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.366087 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.368071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.368045 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.368259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.368242 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.382756 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.382728 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.383190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.383168 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77w88\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.542711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.542672 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:43.860864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.860838 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml"] Oct 30 21:26:43.861968 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:26:43.861944 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e826dbd_077f_4a18_93ef_b5398198f753.slice/crio-f348fc2cd7160179ff66936c28cbe6bc7fbc13bce18869468e0e2979322bdd49 WatchSource:0}: Error finding container f348fc2cd7160179ff66936c28cbe6bc7fbc13bce18869468e0e2979322bdd49: Status 404 returned error can't find the container with id f348fc2cd7160179ff66936c28cbe6bc7fbc13bce18869468e0e2979322bdd49 Oct 30 21:26:43.864441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.864370 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:26:43.864540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.864485 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:26:43.864540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:43.864519 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:26:44.616035 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:44.616004 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" event={"ID":"1e826dbd-077f-4a18-93ef-b5398198f753","Type":"ContainerStarted","Data":"6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c"} Oct 30 21:26:44.616554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:44.616056 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" event={"ID":"1e826dbd-077f-4a18-93ef-b5398198f753","Type":"ContainerStarted","Data":"f348fc2cd7160179ff66936c28cbe6bc7fbc13bce18869468e0e2979322bdd49"} Oct 30 21:26:44.684421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:44.678444 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podStartSLOduration=1.678419476 podStartE2EDuration="1.678419476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:44.665685812 +0000 UTC m=+1043.875625442" watchObservedRunningTime="2026-10-30 21:26:44.678419476 +0000 UTC m=+1043.888359116" Oct 30 21:26:44.728920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:44.728882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:45.543300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:45.543261 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:45.545209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:45.545180 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:26:45.545347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:45.545231 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:26:45.783164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:45.783127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:46.369807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:46.369773 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:46.395291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:46.395260 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:46.532067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:46.532027 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:46.543162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:46.543127 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:26:46.543366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:46.543192 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:26:46.639653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:46.639612 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:47.543159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:47.543123 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:26:47.543592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:47.543196 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:26:48.337554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:48.337522 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:48.543656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:48.543620 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:26:48.544036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:48.543685 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:26:49.543307 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:49.543264 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:26:49.543500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:49.543324 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:26:50.113899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:50.113848 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:50.869691 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:50.869658 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:50.870324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:50.870286 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:50.871352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:50.871329 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:26:51.679646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:51.679607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:53.278782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:53.278749 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:54.649756 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:54.649718 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:26:54.886849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:54.886806 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:55.185409 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:26:55.185357 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:26:55.320401 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:55.320347 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:57.473416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.471872 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 21:26:57.544415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.543511 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:57.544415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.543595 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.556416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.555554 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:26:57.560413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.556863 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.630417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.629511 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.748993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.747559 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.834719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:57.833502 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.055673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.055548 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.056413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.055925 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 21:26:58.145346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.145300 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.486535 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.486486 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 21:26:58.528401 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.528349 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:26:58.573422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.565342 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.589896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.577473 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:26:58.600461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.590170 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:26:58.600754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.600733 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:26:58.742376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.716019 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 21:26:58.743424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.742657 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:58.757459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.756650 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.757459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.757327 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:26:58.906411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.905527 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.906411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.906285 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:26:58.931415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:58.907168 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 21:26:59.081470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.070621 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.185427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.157815 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:26:59.302135 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.302107 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:26:59.302462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.302443 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:26:59.544426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.543880 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 21:26:59.597414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.596216 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 21:26:59.598515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.596196 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:26:59.729172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.729126 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.789935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.789903 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:26:59.790818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.790779 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:59.800685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.800587 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.838496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.838466 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:26:59.916536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:26:59.916484 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.072622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.072519 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 21:27:00.111645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.111614 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:27:00.181442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.168746 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:27:00.544540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.544491 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.612461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.596461 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 21:27:00.740411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.735569 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:27:00.843881 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.833050 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:27:00.846776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.844923 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.896361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:00.894952 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 21:27:01.033894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:01.033325 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 21:27:01.033894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:01.033427 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:27:03.254627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:03.254583 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:03.921816 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:03.921782 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:04.402162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:04.402101 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:04.605117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:04.605077 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:05.184294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:05.184247 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:05.865939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:05.865903 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:06.353255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:06.353217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:06.501375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:06.501337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:07.094291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:07.094249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:08.866830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:08.866798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:09.089969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:09.089910 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:09.333974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:09.333937 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:10.774945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:10.774902 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:10.953914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:10.953881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:11.108494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:11.101062 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:12.240473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.240434 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:12.583690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.583591 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:12.820206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.820166 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5"] Oct 30 21:27:12.821647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.821604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:12.824761 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.824740 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:12.844073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.843998 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5"] Oct 30 21:27:12.868324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.868293 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3f2f4ebe\"" Oct 30 21:27:12.871567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.871537 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3f2f4ebe-dockercfg-p47mt\"" Oct 30 21:27:12.905458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.903185 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:12.905458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:12.903438 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqvg\" (UniqueName: \"kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:13.004640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:13.004605 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqvg\" (UniqueName: \"kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:13.004848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:13.004720 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:13.005743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:13.005703 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:13.022129 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:13.022096 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqvg\" (UniqueName: \"kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:13.141032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:13.141000 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:13.449885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:13.449853 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5"] Oct 30 21:27:13.451203 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:27:13.451176 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b6530d_6419_49a9_9302_75f5400f5871.slice/crio-5020ab62f2dedec7dda66de6f83a787dc131d913f047f196cd1b947e4233c58b WatchSource:0}: Error finding container 5020ab62f2dedec7dda66de6f83a787dc131d913f047f196cd1b947e4233c58b: Status 404 returned error can't find the container with id 5020ab62f2dedec7dda66de6f83a787dc131d913f047f196cd1b947e4233c58b Oct 30 21:27:14.286289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:14.286251 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" event={"ID":"27b6530d-6419-49a9-9302-75f5400f5871","Type":"ContainerStarted","Data":"8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74"} Oct 30 21:27:14.286576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:14.286317 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" event={"ID":"27b6530d-6419-49a9-9302-75f5400f5871","Type":"ContainerStarted","Data":"5020ab62f2dedec7dda66de6f83a787dc131d913f047f196cd1b947e4233c58b"} Oct 30 21:27:14.286576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:14.286351 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:14.323116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:14.323066 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" podStartSLOduration=2.323052518 podStartE2EDuration="2.323052518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:14.319956978 +0000 UTC m=+1073.529896610" watchObservedRunningTime="2026-10-30 21:27:14.323052518 +0000 UTC m=+1073.532992147" Oct 30 21:27:14.386768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:14.386728 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:15.976848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:15.976813 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:17.977027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:17.976992 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:19.745203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:19.745166 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:19.885026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:19.884991 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:21.679017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:21.678955 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:23.207111 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.207076 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6"] Oct 30 21:27:23.212344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.212324 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.236835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.236808 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3f2f4ebe-serving-cert\"" Oct 30 21:27:23.241164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.241136 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6"] Oct 30 21:27:23.273702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.273670 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.273859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.273732 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tzl\" (UniqueName: \"kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.273859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.273789 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.274142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.274115 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.371344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.371314 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:23.375716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.375690 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.375867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.375782 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.375867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.375818 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tzl\" (UniqueName: \"kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.375867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.375863 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.376032 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:27:23.375875 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-serving-cert: secret "maas-api-e2e-mt-3f2f4ebe-serving-cert" not found Oct 30 21:27:23.376032 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:27:23.375945 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls podName:e1056d9a-555b-4a1c-bdc4-f5425bb26ecf nodeName:}" failed. No retries permitted until 2026-10-30 21:27:23.875925435 +0000 UTC m=+1083.085865048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls") pod "maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" (UID: "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf") : secret "maas-api-e2e-mt-3f2f4ebe-serving-cert" not found Oct 30 21:27:23.376581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.376555 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.378250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.378228 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.412757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.412728 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tzl\" (UniqueName: \"kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.584356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.584259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:23.724956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.724918 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:23.882341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.882255 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:23.884894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:23.884864 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:24.125173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:24.125129 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:24.436587 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:24.436503 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6"] Oct 30 21:27:24.460429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:24.460366 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:27:25.178298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:25.178259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:25.186492 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:27:25.186454 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:27:25.295941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:25.295914 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:27:25.366876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:25.366837 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" event={"ID":"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf","Type":"ContainerStarted","Data":"1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144"} Oct 30 21:27:25.367084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:25.366905 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" event={"ID":"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf","Type":"ContainerStarted","Data":"2bc855cffe0fc060ec40015aafc0eb3a27ec26f4f52772f256e7536ccf846ccd"} Oct 30 21:27:25.367084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:25.367064 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:25.403832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:25.403775 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" podStartSLOduration=2.034645594 podStartE2EDuration="2.403761982s" totalImagesPullingTime="369.116388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:25.400999717 +0000 UTC m=+1084.610939349" watchObservedRunningTime="2026-10-30 21:27:25.403761982 +0000 UTC m=+1084.613701642" Oct 30 21:27:26.875141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:26.874861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:28.446281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:28.446247 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:30.185827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.185787 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:30.487625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.487547 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8"] Oct 30 21:27:30.492021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.492001 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.503841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.503817 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9d677d-openshift-default-dockercfg-l57n8\"" Oct 30 21:27:30.518874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.518850 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8"] Oct 30 21:27:30.569003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.568967 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569029 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569109 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569196 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569251 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569269 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569400 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569443 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.569558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.569539 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nllk\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.670612 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670580 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.670787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670642 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.670787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670659 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.670787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670719 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.670787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670754 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670803 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nllk\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670876 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670906 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.670937 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.671064 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.671082 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.671263 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.671367 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.671548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.671529 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.673103 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.673079 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.673509 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.673491 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.687286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.687262 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.687505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.687486 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nllk\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:30.805136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:30.805057 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:31.116701 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:27:31.116671 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f61f7a_3564_46b5_847e_63b642dadf42.slice/crio-4b8dfab6a008c78fdd505c548b858ebaae504ad5fbd31d237a215ad09237a995 WatchSource:0}: Error finding container 4b8dfab6a008c78fdd505c548b858ebaae504ad5fbd31d237a215ad09237a995: Status 404 returned error can't find the container with id 4b8dfab6a008c78fdd505c548b858ebaae504ad5fbd31d237a215ad09237a995 Oct 30 21:27:31.116967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.116936 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8"] Oct 30 21:27:31.119164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.119132 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:27:31.119253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.119194 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:27:31.119253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.119223 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:27:31.378527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.378447 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:27:31.405286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.405254 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" event={"ID":"b3f61f7a-3564-46b5-847e-63b642dadf42","Type":"ContainerStarted","Data":"8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3"} Oct 30 21:27:31.405467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.405317 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" event={"ID":"b3f61f7a-3564-46b5-847e-63b642dadf42","Type":"ContainerStarted","Data":"4b8dfab6a008c78fdd505c548b858ebaae504ad5fbd31d237a215ad09237a995"} Oct 30 21:27:31.450453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.450354 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podStartSLOduration=1.45033415 podStartE2EDuration="1.45033415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:31.447495412 +0000 UTC m=+1090.657435043" watchObservedRunningTime="2026-10-30 21:27:31.45033415 +0000 UTC m=+1090.660273781" Oct 30 21:27:31.805352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.805309 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:31.875586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:31.875554 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:32.807196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:32.807159 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:27:32.807602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:32.807227 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:27:33.480307 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:33.477009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:33.806345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:33.806263 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:27:33.806345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:33.806326 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:27:34.807417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:34.805863 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:27:34.807417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:34.805938 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:27:35.227829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:35.227786 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:35.806192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:35.806158 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:27:35.806371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:35.806222 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:27:35.992796 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:35.992759 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:27:36.805938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:36.805900 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:27:36.806128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:36.805957 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:27:37.328366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:37.328326 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:37.806301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:37.806262 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:27:37.806486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:37.806340 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:27:39.164990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:39.164944 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:39.760716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:39.760687 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:39.761166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:39.761135 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:39.761937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:39.761919 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:27:39.949798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:39.938049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:40.433009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:40.432972 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:40.925165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:40.925131 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:41.208601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:41.208329 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:41.555347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:41.555254 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:41.720781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:41.720740 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:42.575044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.575006 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd"] Oct 30 21:27:42.580091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.580053 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.600591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.600564 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd"] Oct 30 21:27:42.606624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.606597 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3f2f4ebe\"/\"default-dockercfg-mbhg9\"" Oct 30 21:27:42.606764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.606637 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3f2f4ebe\"/\"e2e-lifecycle-model-3f2f4ebe-kserve-self-signed-certs\"" Oct 30 21:27:42.609974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.609951 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3f2f4ebe\"/\"openshift-service-ca.crt\"" Oct 30 21:27:42.610082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.609976 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3f2f4ebe\"/\"kube-root-ca.crt\"" Oct 30 21:27:42.687843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.687800 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:42.756769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.756736 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.756943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.756784 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.757095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.757068 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.757133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.757114 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.757280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.757257 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfsp\" (UniqueName: \"kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.757364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.757341 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.857475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.850011 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:42.860123 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860088 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfsp\" (UniqueName: \"kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.860295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860171 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.860295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860226 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.860295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860259 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.860514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860429 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.860514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860462 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.860980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.860943 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.861660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.861638 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.862595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.862477 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.865241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.865199 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.865773 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.865753 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.883715 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.883503 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfsp\" (UniqueName: \"kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp\") pod \"e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:42.896352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:42.896311 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:43.255485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:43.255446 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd"] Oct 30 21:27:43.258629 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:27:43.258603 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6be7076_59d7_4cf4_96a0_eb9edb608afb.slice/crio-bf15d16f74d005989033b6f9a14dcb5f1f0f9f308a1345fda21f19123b968961 WatchSource:0}: Error finding container bf15d16f74d005989033b6f9a14dcb5f1f0f9f308a1345fda21f19123b968961: Status 404 returned error can't find the container with id bf15d16f74d005989033b6f9a14dcb5f1f0f9f308a1345fda21f19123b968961 Oct 30 21:27:43.495273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:43.495234 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" event={"ID":"a6be7076-59d7-4cf4-96a0-eb9edb608afb","Type":"ContainerStarted","Data":"8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428"} Oct 30 21:27:43.495273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:43.495277 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" event={"ID":"a6be7076-59d7-4cf4-96a0-eb9edb608afb","Type":"ContainerStarted","Data":"bf15d16f74d005989033b6f9a14dcb5f1f0f9f308a1345fda21f19123b968961"} Oct 30 21:27:43.525175 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:43.525104 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:43.532322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:43.532281 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" podStartSLOduration=1.5322665290000002 podStartE2EDuration="1.532266529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:43.529843508 +0000 UTC m=+1102.739783136" watchObservedRunningTime="2026-10-30 21:27:43.532266529 +0000 UTC m=+1102.742206161" Oct 30 21:27:43.896859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:43.896829 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:45.166891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:45.166860 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:45.375637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:45.375608 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:45.610723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:45.610645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:46.862655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:46.862619 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:47.011002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:47.010958 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:47.293011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:47.292971 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:48.177450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:48.177417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:48.335147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:48.335112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:48.891736 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:48.891695 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:50.492988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:50.492953 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:52.082724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:52.082683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:52.897063 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:52.897031 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:52.909874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:52.909848 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:53.567640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:53.567596 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:27:53.674210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:53.674163 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:55.190840 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:27:55.190804 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:27:55.269036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:55.268939 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:55.758512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:55.758474 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:55.937129 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:55.937092 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:56.264718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:56.264679 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:56.450540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:56.450501 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:57.025006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:57.024973 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:58.869721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.869688 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d"] Oct 30 21:27:58.874005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.873987 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:58.893677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.893648 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d"] Oct 30 21:27:58.898969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.898948 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-834924-serving-cert\"" Oct 30 21:27:58.980287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.980251 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:58.980480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.980422 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:58.980569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.980556 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:58.980677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:58.980658 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxht4\" (UniqueName: \"kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.007753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.007716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:59.082186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.082154 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.082378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.082263 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxht4\" (UniqueName: \"kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.082378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.082316 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.082554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.082479 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.082929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.082894 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.084770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.084742 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.084874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.084779 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.098475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.098450 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxht4\" (UniqueName: \"kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4\") pod \"maas-api-e2e-api-b-834924-56948868f6-4zl6d\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.146143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.146101 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:59.188345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.188310 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:27:59.500569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.500534 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d"] Oct 30 21:27:59.500760 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:27:59.500733 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b43a8f1_2f42_4c69_92e9_8b858f27bf38.slice/crio-1c71dc4ec6411a86f53347961a8bbfa48ae10ae53d9e5e9cc925922e8b65f440 WatchSource:0}: Error finding container 1c71dc4ec6411a86f53347961a8bbfa48ae10ae53d9e5e9cc925922e8b65f440: Status 404 returned error can't find the container with id 1c71dc4ec6411a86f53347961a8bbfa48ae10ae53d9e5e9cc925922e8b65f440 Oct 30 21:27:59.608948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:27:59.608906 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" event={"ID":"0b43a8f1-2f42-4c69-92e9-8b858f27bf38","Type":"ContainerStarted","Data":"1c71dc4ec6411a86f53347961a8bbfa48ae10ae53d9e5e9cc925922e8b65f440"} Oct 30 21:28:00.482061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:00.482019 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:00.620306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:00.618714 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" event={"ID":"0b43a8f1-2f42-4c69-92e9-8b858f27bf38","Type":"ContainerStarted","Data":"fa3b2780dfd364e71182dee016a3734fdb9c053f7aa7a29c4546576c80f9cb7e"} Oct 30 21:28:00.620306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:00.620177 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:28:00.624345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:00.624313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:00.656579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:00.656531 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" podStartSLOduration=2.350646336 podStartE2EDuration="2.656516977s" totalImagesPullingTime="305.870641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:00.65603808 +0000 UTC m=+1119.865977711" watchObservedRunningTime="2026-10-30 21:28:00.656516977 +0000 UTC m=+1119.866456606" Oct 30 21:28:01.267302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:01.267250 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.134285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:02.134241 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:02.281574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:02.281514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:03.084901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:03.084818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:04.657750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:04.657713 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:06.281687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:06.281653 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:07.645964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:07.645929 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:28:07.892089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:07.892055 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:09.571658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:09.571622 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:11.093588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:11.093555 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.355865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:12.355829 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:12.600778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:12.600740 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:14.341300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:14.341261 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:15.843594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.843558 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr"] Oct 30 21:28:15.847849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.847831 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:15.863071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.863047 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b13793-serving-cert\"" Oct 30 21:28:15.876694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.876668 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr"] Oct 30 21:28:15.932067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.931982 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:15.932338 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.932306 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pphx\" (UniqueName: \"kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:15.932499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.932472 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:15.932752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:15.932724 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.026490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.026450 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:16.033833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.033810 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.033973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.033860 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.033973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.033893 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pphx\" (UniqueName: \"kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.033973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.033966 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.034520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.034494 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.036426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.036405 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.036515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.036431 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.051549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.051525 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pphx\" (UniqueName: \"kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx\") pod \"maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.162505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.162468 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:16.171680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.171645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:16.472631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.472601 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr"] Oct 30 21:28:16.473052 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:28:16.473028 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978a7bc3_9b81_4ee3_b793_4e8963e67567.slice/crio-0ac0b2f134cc9677d5392ab5a9bdaeac9fa1abe8a4da8e2e42bb06d93818c31a WatchSource:0}: Error finding container 0ac0b2f134cc9677d5392ab5a9bdaeac9fa1abe8a4da8e2e42bb06d93818c31a: Status 404 returned error can't find the container with id 0ac0b2f134cc9677d5392ab5a9bdaeac9fa1abe8a4da8e2e42bb06d93818c31a Oct 30 21:28:16.732788 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:16.732753 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" event={"ID":"978a7bc3-9b81-4ee3-b793-4e8963e67567","Type":"ContainerStarted","Data":"0ac0b2f134cc9677d5392ab5a9bdaeac9fa1abe8a4da8e2e42bb06d93818c31a"} Oct 30 21:28:17.740360 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:17.740318 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" event={"ID":"978a7bc3-9b81-4ee3-b793-4e8963e67567","Type":"ContainerStarted","Data":"ab16b99ba60d817f7801a4ab97dfd4d8933c165f54bbfeece7359f7357e85c75"} Oct 30 21:28:17.740789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:17.740461 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:17.779237 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:17.779188 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" podStartSLOduration=2.4810714689999998 podStartE2EDuration="2.779173649s" totalImagesPullingTime="298.10218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:17.778954508 +0000 UTC m=+1136.988894140" watchObservedRunningTime="2026-10-30 21:28:17.779173649 +0000 UTC m=+1136.989113278" Oct 30 21:28:17.879171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:17.879137 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:18.417410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:18.413259 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:28:19.648303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:19.648270 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:20.564030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:20.563987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:20.711151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:20.711119 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:21.240003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:21.239960 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:21.529192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:21.529101 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:23.526643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:23.526604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:23.752427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:23.752399 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:28:25.185847 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:28:25.185802 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:28:25.383460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:25.383425 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:26.293860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:26.293821 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:26.965870 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:26.965830 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:28.511146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:28.511100 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:30.095904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:30.095862 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:31.697090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:31.697047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.519874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:32.519844 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:32.668569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:32.668528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:33.592263 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:33.592209 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:35.438103 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:35.438055 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:35.923666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:35.923628 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:36.589931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:36.589892 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:36.986448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:36.982669 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt"] Oct 30 21:28:36.988770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:36.988742 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.009887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.009864 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt"] Oct 30 21:28:37.016900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.014715 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c6dab8-openshift-default-dockercfg-6sv4f\"" Oct 30 21:28:37.078821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.078295 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:37.119057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.119004 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.119507 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.119458 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.119819 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.119792 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.119958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.119943 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.120193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.120178 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwr7\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.120373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.120332 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.120511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.120496 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.120746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.120725 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.120900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.120884 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.233718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.233678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:37.265236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265147 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwr7\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265210 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265233 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265288 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265315 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265429 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265500 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265544 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265568 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265794 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.265937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.265927 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.266175 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.266146 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.266280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.266183 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.266426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.266361 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.268672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.268641 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.268833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.268811 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.287413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.286466 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwr7\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.288062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.288031 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token\") pod \"e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.321348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.321311 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:37.368963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.368902 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:37.650375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.650351 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt"] Oct 30 21:28:37.651854 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:28:37.651829 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e1cd19_38e4_4325_9ac4_1294eca4a5c4.slice/crio-690e6df1895e0aaac5cfadddf08d01a878a86f9b774a718ade9ca4997efe78cc WatchSource:0}: Error finding container 690e6df1895e0aaac5cfadddf08d01a878a86f9b774a718ade9ca4997efe78cc: Status 404 returned error can't find the container with id 690e6df1895e0aaac5cfadddf08d01a878a86f9b774a718ade9ca4997efe78cc Oct 30 21:28:37.654413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.654354 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:37.654494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.654440 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:37.654494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.654474 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:37.875342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.875303 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" event={"ID":"11e1cd19-38e4-4325-9ac4-1294eca4a5c4","Type":"ContainerStarted","Data":"134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97"} Oct 30 21:28:37.875583 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.875366 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" event={"ID":"11e1cd19-38e4-4325-9ac4-1294eca4a5c4","Type":"ContainerStarted","Data":"690e6df1895e0aaac5cfadddf08d01a878a86f9b774a718ade9ca4997efe78cc"} Oct 30 21:28:37.916713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:37.916600 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podStartSLOduration=1.916582091 podStartE2EDuration="1.916582091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:37.91185548 +0000 UTC m=+1157.121795110" watchObservedRunningTime="2026-10-30 21:28:37.916582091 +0000 UTC m=+1157.126521719" Oct 30 21:28:38.321992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:38.321899 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:38.327329 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:38.327303 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:38.881996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:38.881960 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:38.882897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:38.882879 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:28:39.099048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:39.096948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:40.369904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:40.369861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:40.514976 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:40.514942 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:40.780895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:40.780864 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:42.111415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:42.110824 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:42.265414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:42.264539 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:43.288416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:43.287687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:45.430178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:45.430138 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:47.640653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:47.640603 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:49.024765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:49.024728 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:49.284769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:49.284667 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:50.890978 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:50.890939 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:52.528575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:52.528542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:53.630660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:53.630624 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:28:54.334482 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:54.334446 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:55.186577 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:28:55.186546 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:28:55.891740 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:55.891709 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:56.550013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:56.549980 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:56.849856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:56.849768 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:56.988239 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:56.988200 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:57.147690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:57.147660 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:57.366973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:57.366935 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:59.068068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:28:59.068034 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:00.776451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:00.776363 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:02.372243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:02.372206 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:02.543362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:02.543324 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:02.793867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:02.793807 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:03.029322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:03.029210 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:03.803196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:03.803161 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:03.947620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:03.947586 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:03.958807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:03.958772 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5"] Oct 30 21:29:04.143564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:04.143532 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:04.290863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:04.290831 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:04.433520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:04.433385 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:06.179798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:06.179757 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:06.660549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:06.660515 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:06.817248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:06.817209 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.016658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:08.016616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.354034 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:08.353931 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.503914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:08.503875 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:08.959960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:08.959913 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" podUID="27b6530d-6419-49a9-9302-75f5400f5871" containerName="payload-processing" containerID="cri-o://8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74" gracePeriod=25 Oct 30 21:29:10.143621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:10.143585 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:11.720332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:11.720300 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:12.498921 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:12.498882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:13.392748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:13.392708 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:14.994637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:14.994592 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:15.294229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:15.294130 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" podUID="27b6530d-6419-49a9-9302-75f5400f5871" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:29:15.584973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:15.584881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:15.729479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:15.729442 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:16.960965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:16.960923 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:18.784464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:18.784427 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:19.193840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:19.193809 5039 generic.go:356] "Generic (PLEG): container finished" podID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerID="134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97" exitCode=0 Oct 30 21:29:19.194038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:19.193874 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" event={"ID":"11e1cd19-38e4-4325-9ac4-1294eca4a5c4","Type":"ContainerDied","Data":"134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97"} Oct 30 21:29:19.194348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:19.194328 5039 scope.go:122] "RemoveContainer" containerID="134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97" Oct 30 21:29:19.195836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:19.195794 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:19.195916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:19.195876 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:19.195916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:19.195910 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:20.203643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:20.203599 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" event={"ID":"11e1cd19-38e4-4325-9ac4-1294eca4a5c4","Type":"ContainerStarted","Data":"d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab"} Oct 30 21:29:20.321973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:20.321933 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:29:20.323070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:20.323043 5039 patch_prober.go:29] interesting pod/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:29:20.323202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:20.323099 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 21:29:20.377418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:20.370381 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:20.907576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:20.907542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:21.050570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:21.050546 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:29:21.052594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:21.052573 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:29:21.322527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:21.322443 5039 patch_prober.go:29] interesting pod/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:29:21.322527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:21.322491 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 21:29:22.004181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:22.004144 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:22.140448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:22.140382 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:22.322785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:22.322695 5039 patch_prober.go:29] interesting pod/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:29:22.322785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:22.322755 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 21:29:22.337938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:22.337908 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:23.322007 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:23.321973 5039 patch_prober.go:29] interesting pod/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:29:23.322203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:23.322025 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 21:29:23.882601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:23.882559 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:29:23.884116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:23.884072 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:24.321892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:24.321860 5039 patch_prober.go:29] interesting pod/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:29:24.322068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:24.321913 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 21:29:25.187784 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:29:25.187746 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:29:25.294517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:25.294483 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" podUID="27b6530d-6419-49a9-9302-75f5400f5871" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:29:25.326832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:25.326801 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:29:25.327905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:25.327884 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:29:25.793920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:25.793886 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:27.390439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:27.390399 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:28.977578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:28.977518 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:29.234885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.234820 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:29:29.272089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.272057 5039 generic.go:356] "Generic (PLEG): container finished" podID="27b6530d-6419-49a9-9302-75f5400f5871" containerID="8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74" exitCode=0 Oct 30 21:29:29.272342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.272107 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" event={"ID":"27b6530d-6419-49a9-9302-75f5400f5871","Type":"ContainerDied","Data":"8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74"} Oct 30 21:29:29.272561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.272131 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" Oct 30 21:29:29.272561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.272422 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5" event={"ID":"27b6530d-6419-49a9-9302-75f5400f5871","Type":"ContainerDied","Data":"5020ab62f2dedec7dda66de6f83a787dc131d913f047f196cd1b947e4233c58b"} Oct 30 21:29:29.272561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.272462 5039 scope.go:122] "RemoveContainer" containerID="8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74" Oct 30 21:29:29.282820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.282798 5039 scope.go:122] "RemoveContainer" containerID="8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74" Oct 30 21:29:29.283138 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:29:29.283105 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74\": container with ID starting with 8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74 not found: ID does not exist" containerID="8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74" Oct 30 21:29:29.283212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.283145 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74"} err="failed to get container status \"8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74\": rpc error: code = NotFound desc = could not find container \"8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74\": container with ID starting with 8e7004b0c2fdae8e02da9ce927107ef62a3479e12d3beeecbb69d147c5f63e74 not found: ID does not exist" Oct 30 21:29:29.347475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.347449 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg\" (UniqueName: \"kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg\") pod \"27b6530d-6419-49a9-9302-75f5400f5871\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " Oct 30 21:29:29.347630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.347526 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume\") pod \"27b6530d-6419-49a9-9302-75f5400f5871\" (UID: \"27b6530d-6419-49a9-9302-75f5400f5871\") " Oct 30 21:29:29.347891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.347868 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume" pod "27b6530d-6419-49a9-9302-75f5400f5871" (UID: "27b6530d-6419-49a9-9302-75f5400f5871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:29.349639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.349612 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg" pod "27b6530d-6419-49a9-9302-75f5400f5871" (UID: "27b6530d-6419-49a9-9302-75f5400f5871"). InnerVolumeSpecName "kube-api-access-vsqvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:29.449097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.449065 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqvg\" (UniqueName: \"kubernetes.io/projected/27b6530d-6419-49a9-9302-75f5400f5871-kube-api-access-vsqvg\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:29.449097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.449093 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b6530d-6419-49a9-9302-75f5400f5871-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:29.612314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.612282 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5"] Oct 30 21:29:29.621675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:29.621649 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-6ppx5"] Oct 30 21:29:30.598697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:30.598662 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:30.926101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:30.926065 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b6530d-6419-49a9-9302-75f5400f5871" path="/var/lib/kubelet/pods/27b6530d-6419-49a9-9302-75f5400f5871/volumes" Oct 30 21:29:31.555304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:31.555237 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:29:32.315760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:32.315712 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:33.983410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:33.983212 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:29:33.983410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:33.983298 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:29:34.019624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:33.994052 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:34.195375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:34.195337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:34.209698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:34.209588 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:29:34.210025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:34.209976 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:29:34.637797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:34.637749 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:36.857784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:36.857742 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:38.586334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:38.586293 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:39.884635 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:39.884595 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6"] Oct 30 21:29:39.885116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:39.884842 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" podUID="e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" containerName="maas-api" containerID="cri-o://1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144" gracePeriod=30 Oct 30 21:29:39.996541 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:39.996504 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:40.246415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.242935 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:29:40.359247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.359209 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\") pod \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " Oct 30 21:29:40.359492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.359300 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl\" (UniqueName: \"kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl\") pod \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " Oct 30 21:29:40.359492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.359374 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs\") pod \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " Oct 30 21:29:40.359492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.359453 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca\") pod \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\" (UID: \"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf\") " Oct 30 21:29:40.360468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.360438 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca" pod "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" (UID: "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:40.362434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.362292 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs" pod "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" (UID: "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:40.372802 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.372775 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls" pod "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" (UID: "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:40.372988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.372873 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl" pod "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" (UID: "e1056d9a-555b-4a1c-bdc4-f5425bb26ecf"). InnerVolumeSpecName "kube-api-access-z2tzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:40.393933 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.393894 5039 generic.go:356] "Generic (PLEG): container finished" podID="e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" containerID="1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144" exitCode=0 Oct 30 21:29:40.394122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.393948 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" Oct 30 21:29:40.394122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.393978 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" event={"ID":"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf","Type":"ContainerDied","Data":"1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144"} Oct 30 21:29:40.394122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.394047 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6" event={"ID":"e1056d9a-555b-4a1c-bdc4-f5425bb26ecf","Type":"ContainerDied","Data":"2bc855cffe0fc060ec40015aafc0eb3a27ec26f4f52772f256e7536ccf846ccd"} Oct 30 21:29:40.394122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.394082 5039 scope.go:122] "RemoveContainer" containerID="1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144" Oct 30 21:29:40.405700 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.405683 5039 scope.go:122] "RemoveContainer" containerID="1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144" Oct 30 21:29:40.405976 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:29:40.405953 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144\": container with ID starting with 1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144 not found: ID does not exist" containerID="1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144" Oct 30 21:29:40.406065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.405982 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144"} err="failed to get container status \"1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144\": rpc error: code = NotFound desc = could not find container \"1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144\": container with ID starting with 1d47a37c17753e4a5a95362e58d41ab47e2fa29cccbbd7d4a7bab88016455144 not found: ID does not exist" Oct 30 21:29:40.438016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.437990 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6"] Oct 30 21:29:40.443235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.443213 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-7btw6"] Oct 30 21:29:40.462116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.462072 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:40.462116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.462107 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tzl\" (UniqueName: \"kubernetes.io/projected/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-kube-api-access-z2tzl\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:40.462330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.462161 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:40.462330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.462178 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:40.923241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.923204 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" path="/var/lib/kubelet/pods/e1056d9a-555b-4a1c-bdc4-f5425bb26ecf/volumes" Oct 30 21:29:40.960060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:40.960026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:41.103807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:41.103762 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:41.240672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:41.240578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:41.377986 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:41.377946 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:42.089219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:42.089181 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:42.745012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:42.744964 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:43.091060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:43.090959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:43.238380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:43.238346 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:43.380236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:43.380139 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:44.704331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:44.704294 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:45.351112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:45.351074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:46.212275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.212189 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d"] Oct 30 21:29:46.212806 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.212442 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" podUID="0b43a8f1-2f42-4c69-92e9-8b858f27bf38" containerName="maas-api" containerID="cri-o://fa3b2780dfd364e71182dee016a3734fdb9c053f7aa7a29c4546576c80f9cb7e" gracePeriod=30 Oct 30 21:29:46.333628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.333597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:46.444515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.441228 5039 generic.go:356] "Generic (PLEG): container finished" podID="0b43a8f1-2f42-4c69-92e9-8b858f27bf38" containerID="fa3b2780dfd364e71182dee016a3734fdb9c053f7aa7a29c4546576c80f9cb7e" exitCode=0 Oct 30 21:29:46.444515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.441491 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" event={"ID":"0b43a8f1-2f42-4c69-92e9-8b858f27bf38","Type":"ContainerDied","Data":"fa3b2780dfd364e71182dee016a3734fdb9c053f7aa7a29c4546576c80f9cb7e"} Oct 30 21:29:46.541814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.541786 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:29:46.566777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.566745 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs\") pod \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " Oct 30 21:29:46.566964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.566793 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls\") pod \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " Oct 30 21:29:46.566964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.566851 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4\" (UniqueName: \"kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4\") pod \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " Oct 30 21:29:46.566964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.566906 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca\") pod \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\" (UID: \"0b43a8f1-2f42-4c69-92e9-8b858f27bf38\") " Oct 30 21:29:46.567833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.567804 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca" pod "0b43a8f1-2f42-4c69-92e9-8b858f27bf38" (UID: "0b43a8f1-2f42-4c69-92e9-8b858f27bf38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:46.570447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.570383 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4" pod "0b43a8f1-2f42-4c69-92e9-8b858f27bf38" (UID: "0b43a8f1-2f42-4c69-92e9-8b858f27bf38"). InnerVolumeSpecName "kube-api-access-bxht4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:46.572567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.572541 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs" pod "0b43a8f1-2f42-4c69-92e9-8b858f27bf38" (UID: "0b43a8f1-2f42-4c69-92e9-8b858f27bf38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:46.577084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.577058 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls" pod "0b43a8f1-2f42-4c69-92e9-8b858f27bf38" (UID: "0b43a8f1-2f42-4c69-92e9-8b858f27bf38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:46.669118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.669074 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:46.669118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.669120 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxht4\" (UniqueName: \"kubernetes.io/projected/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-kube-api-access-bxht4\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:46.669353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.669136 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:46.669353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:46.669178 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b43a8f1-2f42-4c69-92e9-8b858f27bf38-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:29:47.303502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.303412 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:47.452290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.452259 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" Oct 30 21:29:47.452575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.452287 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d" event={"ID":"0b43a8f1-2f42-4c69-92e9-8b858f27bf38","Type":"ContainerDied","Data":"1c71dc4ec6411a86f53347961a8bbfa48ae10ae53d9e5e9cc925922e8b65f440"} Oct 30 21:29:47.452575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.452354 5039 scope.go:122] "RemoveContainer" containerID="fa3b2780dfd364e71182dee016a3734fdb9c053f7aa7a29c4546576c80f9cb7e" Oct 30 21:29:47.485178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.485146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:47.510146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.510119 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d"] Oct 30 21:29:47.518605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.518582 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-834924-56948868f6-4zl6d"] Oct 30 21:29:47.641467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:47.641386 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:48.012130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:48.012042 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:48.191814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:48.191775 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:48.922400 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:48.922357 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b43a8f1-2f42-4c69-92e9-8b858f27bf38" path="/var/lib/kubelet/pods/0b43a8f1-2f42-4c69-92e9-8b858f27bf38/volumes" Oct 30 21:29:49.678322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.678289 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:49.835369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.835331 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:49.937544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.937457 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j"] Oct 30 21:29:49.938018 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938005 5039 cpu_manager.go:414] "containerMap: removing container" podUID="0b43a8f1-2f42-4c69-92e9-8b858f27bf38" containerName="maas-api" Oct 30 21:29:49.938068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938019 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b43a8f1-2f42-4c69-92e9-8b858f27bf38" containerName="maas-api" Oct 30 21:29:49.938068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938044 5039 cpu_manager.go:414] "containerMap: removing container" podUID="e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" containerName="maas-api" Oct 30 21:29:49.938068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938049 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" containerName="maas-api" Oct 30 21:29:49.938172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938077 5039 cpu_manager.go:414] "containerMap: removing container" podUID="27b6530d-6419-49a9-9302-75f5400f5871" containerName="payload-processing" Oct 30 21:29:49.938172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938083 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b6530d-6419-49a9-9302-75f5400f5871" containerName="payload-processing" Oct 30 21:29:49.938241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938173 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1056d9a-555b-4a1c-bdc4-f5425bb26ecf" containerName="maas-api" Oct 30 21:29:49.938241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938189 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b6530d-6419-49a9-9302-75f5400f5871" containerName="payload-processing" Oct 30 21:29:49.938241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.938198 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b43a8f1-2f42-4c69-92e9-8b858f27bf38" containerName="maas-api" Oct 30 21:29:49.942861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.942841 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:49.962146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.962118 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j"] Oct 30 21:29:49.966862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.966836 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-342fc9\"" Oct 30 21:29:49.967025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.966943 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-342fc9-dockercfg-q2vkw\"" Oct 30 21:29:49.988538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:49.988509 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:50.014184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.014144 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.014404 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.014263 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78lj\" (UniqueName: \"kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj\") pod \"payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.026325 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.026286 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542"] Oct 30 21:29:50.064760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.064718 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.093853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.093817 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542"] Oct 30 21:29:50.122441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.117988 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78lj\" (UniqueName: \"kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj\") pod \"payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.122441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.118098 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.122441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.118225 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlsfh\" (UniqueName: \"kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh\") pod \"payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.122441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.118336 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.122441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.119437 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.136872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.136841 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78lj\" (UniqueName: \"kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj\") pod \"payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.225463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.222489 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.225463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.222637 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsfh\" (UniqueName: \"kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh\") pod \"payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.225463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.224083 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.243007 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.242963 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlsfh\" (UniqueName: \"kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh\") pod \"payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.256519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.256494 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:50.396180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.396143 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:50.568313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.568173 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j"] Oct 30 21:29:50.568838 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:29:50.568813 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8a7702_1a82_43e8_8f0c_d4bcaa110e5d.slice/crio-22aa91caa31892ec0efcf8a1dba9a45c54e76971b0382d67863db4dc9430b5a8 WatchSource:0}: Error finding container 22aa91caa31892ec0efcf8a1dba9a45c54e76971b0382d67863db4dc9430b5a8: Status 404 returned error can't find the container with id 22aa91caa31892ec0efcf8a1dba9a45c54e76971b0382d67863db4dc9430b5a8 Oct 30 21:29:50.696741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.696717 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542"] Oct 30 21:29:50.697291 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:29:50.697265 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560f36b7_e525_4c67_b0b8_3b6bc794bcc4.slice/crio-1fc368277a27019b43ecde66903e3593803d9a9a97dfadde3725e4780981c878 WatchSource:0}: Error finding container 1fc368277a27019b43ecde66903e3593803d9a9a97dfadde3725e4780981c878: Status 404 returned error can't find the container with id 1fc368277a27019b43ecde66903e3593803d9a9a97dfadde3725e4780981c878 Oct 30 21:29:50.752658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.752627 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:50.919936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:50.919895 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:51.482938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.482900 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" event={"ID":"560f36b7-e525-4c67-b0b8-3b6bc794bcc4","Type":"ContainerStarted","Data":"03ecbc383005df1fe972c1de43ea2cca50a0c33704f4e3f4a97e921db1c291a0"} Oct 30 21:29:51.483342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.482953 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" event={"ID":"560f36b7-e525-4c67-b0b8-3b6bc794bcc4","Type":"ContainerStarted","Data":"1fc368277a27019b43ecde66903e3593803d9a9a97dfadde3725e4780981c878"} Oct 30 21:29:51.483342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.483066 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:29:51.484571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.484540 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" event={"ID":"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d","Type":"ContainerStarted","Data":"f44701ff9f6c1141d93ba759b79cb1d5b3525518fd5b38b1e561f179e459474a"} Oct 30 21:29:51.484694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.484591 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" event={"ID":"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d","Type":"ContainerStarted","Data":"22aa91caa31892ec0efcf8a1dba9a45c54e76971b0382d67863db4dc9430b5a8"} Oct 30 21:29:51.484694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.484622 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:29:51.518203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.518157 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" podStartSLOduration=2.518144444 podStartE2EDuration="2.518144444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:51.517352138 +0000 UTC m=+1230.727291767" watchObservedRunningTime="2026-10-30 21:29:51.518144444 +0000 UTC m=+1230.728084103" Oct 30 21:29:51.557194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.557150 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" podStartSLOduration=2.557138002 podStartE2EDuration="2.557138002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:51.553407629 +0000 UTC m=+1230.763347259" watchObservedRunningTime="2026-10-30 21:29:51.557138002 +0000 UTC m=+1230.767077632" Oct 30 21:29:51.778971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:51.778869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:52.358955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:52.358919 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:52.572212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:52.572174 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:53.205126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.205082 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:53.346822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.346786 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97"] Oct 30 21:29:53.354488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.354459 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.374747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.374721 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97"] Oct 30 21:29:53.380108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.380084 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3f2f4ebe\"" Oct 30 21:29:53.380242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.380172 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3f2f4ebe-dockercfg-pwbgv\"" Oct 30 21:29:53.496515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.496238 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.496515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.496315 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkk5l\" (UniqueName: \"kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.514228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.514167 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj"] Oct 30 21:29:53.519366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.519329 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj"] Oct 30 21:29:53.519537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.519459 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.597660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.597621 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.598088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.597722 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n2h\" (UniqueName: \"kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.598088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.597799 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.598088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.597830 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkk5l\" (UniqueName: \"kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.598593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.598574 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.616160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.616131 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkk5l\" (UniqueName: \"kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.669857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.669824 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:53.698986 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.698901 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.699124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.699011 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n2h\" (UniqueName: \"kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.700294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.700269 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.720379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.720318 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n2h\" (UniqueName: \"kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h\") pod \"payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.844038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.844008 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:53.998705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:53.998647 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97"] Oct 30 21:29:53.998990 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:29:53.998966 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e77cc5_aa29_4154_ba46_f476d8c9d1fe.slice/crio-2ac49b13e59116001d8d51f0c53253077c4dd0738ce9bceb90e17a65947b354c WatchSource:0}: Error finding container 2ac49b13e59116001d8d51f0c53253077c4dd0738ce9bceb90e17a65947b354c: Status 404 returned error can't find the container with id 2ac49b13e59116001d8d51f0c53253077c4dd0738ce9bceb90e17a65947b354c Oct 30 21:29:54.151185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.151159 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj"] Oct 30 21:29:54.154210 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:29:54.154184 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251c5c30_0b6b_4d86_a3f7_2ca7846ce65e.slice/crio-0f3e0d3cd18e4bf11cdb67eb55b42f49de5273f9c96ddd20b0bfc5b79fd2696e WatchSource:0}: Error finding container 0f3e0d3cd18e4bf11cdb67eb55b42f49de5273f9c96ddd20b0bfc5b79fd2696e: Status 404 returned error can't find the container with id 0f3e0d3cd18e4bf11cdb67eb55b42f49de5273f9c96ddd20b0bfc5b79fd2696e Oct 30 21:29:54.524096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.524057 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" event={"ID":"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e","Type":"ContainerStarted","Data":"5f346d1818a66757aba31a8e0dbf9768bad2eea8b4d45ab3e625c428349fcf56"} Oct 30 21:29:54.524267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.524123 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" event={"ID":"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e","Type":"ContainerStarted","Data":"0f3e0d3cd18e4bf11cdb67eb55b42f49de5273f9c96ddd20b0bfc5b79fd2696e"} Oct 30 21:29:54.524267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.524151 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:29:54.525971 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.525943 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" event={"ID":"22e77cc5-aa29-4154-ba46-f476d8c9d1fe","Type":"ContainerStarted","Data":"ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9"} Oct 30 21:29:54.526102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.525993 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:29:54.526102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.526012 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" event={"ID":"22e77cc5-aa29-4154-ba46-f476d8c9d1fe","Type":"ContainerStarted","Data":"2ac49b13e59116001d8d51f0c53253077c4dd0738ce9bceb90e17a65947b354c"} Oct 30 21:29:54.559001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.558939 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" podStartSLOduration=1.5589231209999999 podStartE2EDuration="1.558923121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:54.558025151 +0000 UTC m=+1233.767964794" watchObservedRunningTime="2026-10-30 21:29:54.558923121 +0000 UTC m=+1233.768862752" Oct 30 21:29:54.601291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:54.601243 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" podStartSLOduration=1.6012290500000002 podStartE2EDuration="1.60122905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:54.59657579 +0000 UTC m=+1233.806515420" watchObservedRunningTime="2026-10-30 21:29:54.60122905 +0000 UTC m=+1233.811168677" Oct 30 21:29:55.185062 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:29:55.185023 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:29:55.935777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:55.935744 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:56.106046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:56.106009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:56.994999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:56.991532 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:29:57.913537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.913497 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql"] Oct 30 21:29:57.929741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.929714 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:57.934662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.934630 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql"] Oct 30 21:29:57.948145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.948121 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3f2f4ebe-serving-cert\"" Oct 30 21:29:57.961111 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.961078 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:57.961371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.961351 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:57.961504 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.961491 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:57.961807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.961788 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65476\" (UniqueName: \"kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:57.997338 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:57.997306 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:58.063068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.063036 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.063233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.063078 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.063233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.063099 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.063233 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:29:58.063206 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-serving-cert: secret "maas-api-e2e-mt-3f2f4ebe-serving-cert" not found Oct 30 21:29:58.063341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.063267 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65476\" (UniqueName: \"kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.063414 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:29:58.063385 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls podName:57bfdae8-8de2-4f34-8f77-9b3f550f458e nodeName:}" failed. No retries permitted until 2026-10-30 21:29:58.563363073 +0000 UTC m=+1237.773302682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls") pod "maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" (UID: "57bfdae8-8de2-4f34-8f77-9b3f550f458e") : secret "maas-api-e2e-mt-3f2f4ebe-serving-cert" not found Oct 30 21:29:58.063829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.063807 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.065672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.065652 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.081893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.081868 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65476\" (UniqueName: \"kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.142605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.142568 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:58.570689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.570650 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.573213 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.573190 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\") pod \"maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.590012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.589981 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:58.894000 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:58.893965 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql"] Oct 30 21:29:58.897497 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:29:58.897466 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bfdae8_8de2_4f34_8f77_9b3f550f458e.slice/crio-6521443ef7e04ed06df05a5d15843f9a6167d8b3907e3cbcf1ba4e6e97d908ae WatchSource:0}: Error finding container 6521443ef7e04ed06df05a5d15843f9a6167d8b3907e3cbcf1ba4e6e97d908ae: Status 404 returned error can't find the container with id 6521443ef7e04ed06df05a5d15843f9a6167d8b3907e3cbcf1ba4e6e97d908ae Oct 30 21:29:59.298279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:59.298248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:59.559735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:59.559644 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" event={"ID":"57bfdae8-8de2-4f34-8f77-9b3f550f458e","Type":"ContainerStarted","Data":"c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5"} Oct 30 21:29:59.559735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:59.559708 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" event={"ID":"57bfdae8-8de2-4f34-8f77-9b3f550f458e","Type":"ContainerStarted","Data":"6521443ef7e04ed06df05a5d15843f9a6167d8b3907e3cbcf1ba4e6e97d908ae"} Oct 30 21:29:59.560337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:59.559738 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:29:59.599504 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:59.599446 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" podStartSLOduration=2.2279244 podStartE2EDuration="2.5994279s" totalImagesPullingTime="371.5035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:59.594867726 +0000 UTC m=+1238.804807357" watchObservedRunningTime="2026-10-30 21:29:59.5994279 +0000 UTC m=+1238.809367549" Oct 30 21:29:59.993325 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:29:59.993278 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:00.186838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.185699 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx"] Oct 30 21:30:00.191021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.190628 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.191467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.191439 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf"] Oct 30 21:30:00.196405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.196333 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc"] Oct 30 21:30:00.196519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.196473 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.201301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.201279 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.208975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.208951 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf"] Oct 30 21:30:00.214644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.214584 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx"] Oct 30 21:30:00.218556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.217105 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc"] Oct 30 21:30:00.285354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.285328 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\"" Oct 30 21:30:00.300962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.300937 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.301303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.300979 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jdp\" (UniqueName: \"kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.301303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.301103 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.301303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.301200 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.301303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.301241 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2sz\" (UniqueName: \"kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.301544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.301413 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5qd\" (UniqueName: \"kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd\") pod \"maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.402220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.402188 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.402405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.402243 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2sz\" (UniqueName: \"kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.402405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.402295 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5qd\" (UniqueName: \"kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd\") pod \"maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.402405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.402356 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.402582 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.402384 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jdp\" (UniqueName: \"kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.402582 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.402507 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.403136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.403114 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.403136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.403130 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.403242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.403205 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.419094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.419063 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5qd\" (UniqueName: \"kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd\") pod \"maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.419184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.419063 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2sz\" (UniqueName: \"kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.419184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.419116 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jdp\" (UniqueName: \"kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.551702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.551611 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:00.558010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.557977 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:00.569627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.567185 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:00.876670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.876635 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf"] Oct 30 21:30:00.877479 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:30:00.877452 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b430541_0361_4df9_8ce5_be25f8d44e7c.slice/crio-7b101a503eb8fbe8180f0afb7b278c97b8270621fb8c55d75801b4b17fa668fd WatchSource:0}: Error finding container 7b101a503eb8fbe8180f0afb7b278c97b8270621fb8c55d75801b4b17fa668fd: Status 404 returned error can't find the container with id 7b101a503eb8fbe8180f0afb7b278c97b8270621fb8c55d75801b4b17fa668fd Oct 30 21:30:00.886864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.886835 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx"] Oct 30 21:30:00.887349 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:30:00.887287 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf354f2c2_c97a_4083_8ae8_db2b840d9283.slice/crio-110c192e51471e29bbcaa6a0f5c516d6948421db6f00d4963829dcd4d8faee1d WatchSource:0}: Error finding container 110c192e51471e29bbcaa6a0f5c516d6948421db6f00d4963829dcd4d8faee1d: Status 404 returned error can't find the container with id 110c192e51471e29bbcaa6a0f5c516d6948421db6f00d4963829dcd4d8faee1d Oct 30 21:30:00.890678 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:30:00.890208 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619994dc_cf66_4d77_a6bd_7910bc013b63.slice/crio-e028988b71da81cc42217f818e9e943ccda410464f3959288004c309bb27745b WatchSource:0}: Error finding container e028988b71da81cc42217f818e9e943ccda410464f3959288004c309bb27745b: Status 404 returned error can't find the container with id e028988b71da81cc42217f818e9e943ccda410464f3959288004c309bb27745b Oct 30 21:30:00.890968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:00.890945 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc"] Oct 30 21:30:01.582078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:01.582045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:01.593632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:01.593596 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerStarted","Data":"110c192e51471e29bbcaa6a0f5c516d6948421db6f00d4963829dcd4d8faee1d"} Oct 30 21:30:01.597197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:01.596933 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerStarted","Data":"7b101a503eb8fbe8180f0afb7b278c97b8270621fb8c55d75801b4b17fa668fd"} Oct 30 21:30:01.599082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:01.599050 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerStarted","Data":"e028988b71da81cc42217f818e9e943ccda410464f3959288004c309bb27745b"} Oct 30 21:30:01.738865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:01.738820 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:01.899316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:01.899273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:02.496735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:02.496661 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:30:02.498258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:02.498229 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:30:03.368709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:03.368672 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:04.627265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.627172 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerStarted","Data":"baa60e2ed97d1e57afec0a3ca483aef8f329271c2c439ff1d2a5533e54d28129"} Oct 30 21:30:04.628936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.628905 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerStarted","Data":"f2c10e5855c2fd3dbf1ba6e0cdbe057c8e78d7090deb67a160ac1e9708f95c36"} Oct 30 21:30:04.630535 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.630514 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerStarted","Data":"d6ddd6b183a224153fcb70b7c11af8cd3301113b090a1a6554b22e6cf3714760"} Oct 30 21:30:04.699724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.699663 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" podStartSLOduration=1.763575321 podStartE2EDuration="4.6996478s" totalImagesPullingTime="2.936072479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.659830605 +0000 UTC m=+1243.869770237" watchObservedRunningTime="2026-10-30 21:30:04.6996478 +0000 UTC m=+1243.909587429" Oct 30 21:30:04.700043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.700001 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" podStartSLOduration=1.723286882 podStartE2EDuration="4.699991417s" totalImagesPullingTime="2.976704535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.696428156 +0000 UTC m=+1243.906367814" watchObservedRunningTime="2026-10-30 21:30:04.699991417 +0000 UTC m=+1243.909931051" Oct 30 21:30:04.733244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.733189 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" podStartSLOduration=1.770414573 podStartE2EDuration="4.733173941s" totalImagesPullingTime="2.962759368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.729620549 +0000 UTC m=+1243.939560179" watchObservedRunningTime="2026-10-30 21:30:04.733173941 +0000 UTC m=+1243.943113571" Oct 30 21:30:04.911318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.911292 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp"] Oct 30 21:30:04.915776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.915754 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:04.933650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.933627 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-342fc9-serving-cert\"" Oct 30 21:30:04.937575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.937552 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp"] Oct 30 21:30:04.974057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.974027 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:04.974172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.974147 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:04.974344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.974325 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:04.974529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:04.974513 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpm4\" (UniqueName: \"kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.050468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.050431 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:05.075958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.075930 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpm4\" (UniqueName: \"kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.076083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.076038 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.076140 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.076091 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.076212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.076196 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.076344 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:05.076326 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-serving-cert: secret "maas-api-e2e-api-b-342fc9-serving-cert" not found Oct 30 21:30:05.076430 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:05.076416 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls podName:7063944a-7a60-48d0-953f-ebcfbdf20f92 nodeName:}" failed. No retries permitted until 2026-10-30 21:30:05.57637859 +0000 UTC m=+1244.786318214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls") pod "maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" (UID: "7063944a-7a60-48d0-953f-ebcfbdf20f92") : secret "maas-api-e2e-api-b-342fc9-serving-cert" not found Oct 30 21:30:05.076832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.076808 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.078479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.078458 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.095075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.095047 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpm4\" (UniqueName: \"kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.212524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.212412 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:05.361084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.361039 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:05.536208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.536126 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:30:05.536208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.536205 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:30:05.582580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.582549 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.590335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.590303 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:30:05.593199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.593171 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\") pod \"maas-api-e2e-api-b-342fc9-6467758b99-rrcwp\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:05.643202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.643164 5039 generic.go:356] "Generic (PLEG): container finished" podID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerID="f2c10e5855c2fd3dbf1ba6e0cdbe057c8e78d7090deb67a160ac1e9708f95c36" exitCode=60 Oct 30 21:30:05.643730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.643255 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerDied","Data":"f2c10e5855c2fd3dbf1ba6e0cdbe057c8e78d7090deb67a160ac1e9708f95c36"} Oct 30 21:30:05.643730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.643562 5039 scope.go:122] "RemoveContainer" containerID="f2c10e5855c2fd3dbf1ba6e0cdbe057c8e78d7090deb67a160ac1e9708f95c36" Oct 30 21:30:05.645323 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.645295 5039 generic.go:356] "Generic (PLEG): container finished" podID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerID="d6ddd6b183a224153fcb70b7c11af8cd3301113b090a1a6554b22e6cf3714760" exitCode=7 Oct 30 21:30:05.646034 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.645982 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerDied","Data":"d6ddd6b183a224153fcb70b7c11af8cd3301113b090a1a6554b22e6cf3714760"} Oct 30 21:30:05.646246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.646229 5039 scope.go:122] "RemoveContainer" containerID="d6ddd6b183a224153fcb70b7c11af8cd3301113b090a1a6554b22e6cf3714760" Oct 30 21:30:05.648599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.648577 5039 generic.go:356] "Generic (PLEG): container finished" podID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerID="baa60e2ed97d1e57afec0a3ca483aef8f329271c2c439ff1d2a5533e54d28129" exitCode=60 Oct 30 21:30:05.648787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.648638 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerDied","Data":"baa60e2ed97d1e57afec0a3ca483aef8f329271c2c439ff1d2a5533e54d28129"} Oct 30 21:30:05.649026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.648974 5039 scope.go:122] "RemoveContainer" containerID="baa60e2ed97d1e57afec0a3ca483aef8f329271c2c439ff1d2a5533e54d28129" Oct 30 21:30:05.855019 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:05.854934 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:06.237653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.237620 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp"] Oct 30 21:30:06.657993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.657959 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerStarted","Data":"ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632"} Oct 30 21:30:06.659701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.659555 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" event={"ID":"7063944a-7a60-48d0-953f-ebcfbdf20f92","Type":"ContainerStarted","Data":"656c69b0056da2a82e79a8550c151549d6b2978734aa9cf6278ef74a0c4f5783"} Oct 30 21:30:06.661798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.661760 5039 generic.go:356] "Generic (PLEG): container finished" podID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerID="cc0173c3960e948bed5bcacaae99a1f8a97e31bb253ca1aab6381cd3d1af1e0f" exitCode=60 Oct 30 21:30:06.661903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.661815 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerDied","Data":"cc0173c3960e948bed5bcacaae99a1f8a97e31bb253ca1aab6381cd3d1af1e0f"} Oct 30 21:30:06.661903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.661888 5039 scope.go:122] "RemoveContainer" containerID="d6ddd6b183a224153fcb70b7c11af8cd3301113b090a1a6554b22e6cf3714760" Oct 30 21:30:06.662124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.662098 5039 scope.go:122] "RemoveContainer" containerID="cc0173c3960e948bed5bcacaae99a1f8a97e31bb253ca1aab6381cd3d1af1e0f" Oct 30 21:30:06.662439 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:06.662413 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc_odh-ai-gateway-infra(619994dc-cf66-4d77-a6bd-7910bc013b63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" Oct 30 21:30:06.666182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.666157 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerStarted","Data":"51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad"} Oct 30 21:30:06.972507 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:06.972471 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:07.673701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.673670 5039 generic.go:356] "Generic (PLEG): container finished" podID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerID="51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad" exitCode=60 Oct 30 21:30:07.674212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.673744 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerDied","Data":"51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad"} Oct 30 21:30:07.674212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.673798 5039 scope.go:122] "RemoveContainer" containerID="baa60e2ed97d1e57afec0a3ca483aef8f329271c2c439ff1d2a5533e54d28129" Oct 30 21:30:07.674212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.673963 5039 scope.go:122] "RemoveContainer" containerID="51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad" Oct 30 21:30:07.674361 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:07.674228 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx_odh-ai-gateway-infra(f354f2c2-c97a-4083-8ae8-db2b840d9283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" Oct 30 21:30:07.675955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.675932 5039 generic.go:356] "Generic (PLEG): container finished" podID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerID="ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632" exitCode=60 Oct 30 21:30:07.676062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.676014 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerDied","Data":"ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632"} Oct 30 21:30:07.676209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.676195 5039 scope.go:122] "RemoveContainer" containerID="ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632" Oct 30 21:30:07.676475 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:07.676444 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf_odh-ai-gateway-infra(9b430541-0361-4df9-8ce5-be25f8d44e7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" Oct 30 21:30:07.677624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.677601 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" event={"ID":"7063944a-7a60-48d0-953f-ebcfbdf20f92","Type":"ContainerStarted","Data":"409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62"} Oct 30 21:30:07.677721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.677658 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:07.679514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.679495 5039 scope.go:122] "RemoveContainer" containerID="cc0173c3960e948bed5bcacaae99a1f8a97e31bb253ca1aab6381cd3d1af1e0f" Oct 30 21:30:07.679680 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:07.679662 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc_odh-ai-gateway-infra(619994dc-cf66-4d77-a6bd-7910bc013b63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" Oct 30 21:30:07.686278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.686251 5039 scope.go:122] "RemoveContainer" containerID="f2c10e5855c2fd3dbf1ba6e0cdbe057c8e78d7090deb67a160ac1e9708f95c36" Oct 30 21:30:07.803417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:07.803355 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" podStartSLOduration=3.491158924 podStartE2EDuration="3.803341731s" totalImagesPullingTime="312.182807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:07.800133415 +0000 UTC m=+1247.010073045" watchObservedRunningTime="2026-10-30 21:30:07.803341731 +0000 UTC m=+1247.013281360" Oct 30 21:30:08.433444 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.433407 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr"] Oct 30 21:30:08.433839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.433680 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" podUID="978a7bc3-9b81-4ee3-b793-4e8963e67567" containerName="maas-api" containerID="cri-o://ab16b99ba60d817f7801a4ab97dfd4d8933c165f54bbfeece7359f7357e85c75" gracePeriod=30 Oct 30 21:30:08.549266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.549227 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:08.729684 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.728603 5039 scope.go:122] "RemoveContainer" containerID="51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad" Oct 30 21:30:08.729684 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:08.728988 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx_odh-ai-gateway-infra(f354f2c2-c97a-4083-8ae8-db2b840d9283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" Oct 30 21:30:08.732699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.732660 5039 scope.go:122] "RemoveContainer" containerID="ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632" Oct 30 21:30:08.733014 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:08.732987 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf_odh-ai-gateway-infra(9b430541-0361-4df9-8ce5-be25f8d44e7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" Oct 30 21:30:08.735422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.735379 5039 generic.go:356] "Generic (PLEG): container finished" podID="978a7bc3-9b81-4ee3-b793-4e8963e67567" containerID="ab16b99ba60d817f7801a4ab97dfd4d8933c165f54bbfeece7359f7357e85c75" exitCode=0 Oct 30 21:30:08.736359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.736284 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" event={"ID":"978a7bc3-9b81-4ee3-b793-4e8963e67567","Type":"ContainerDied","Data":"ab16b99ba60d817f7801a4ab97dfd4d8933c165f54bbfeece7359f7357e85c75"} Oct 30 21:30:08.820409 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.819487 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:30:08.940295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.940260 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs\") pod \"978a7bc3-9b81-4ee3-b793-4e8963e67567\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " Oct 30 21:30:08.940500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.940357 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx\" (UniqueName: \"kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx\") pod \"978a7bc3-9b81-4ee3-b793-4e8963e67567\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " Oct 30 21:30:08.940500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.940443 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls\") pod \"978a7bc3-9b81-4ee3-b793-4e8963e67567\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " Oct 30 21:30:08.940500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.940472 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca\") pod \"978a7bc3-9b81-4ee3-b793-4e8963e67567\" (UID: \"978a7bc3-9b81-4ee3-b793-4e8963e67567\") " Oct 30 21:30:08.941371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.941337 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca" pod "978a7bc3-9b81-4ee3-b793-4e8963e67567" (UID: "978a7bc3-9b81-4ee3-b793-4e8963e67567"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:08.945871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.945821 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx" pod "978a7bc3-9b81-4ee3-b793-4e8963e67567" (UID: "978a7bc3-9b81-4ee3-b793-4e8963e67567"). InnerVolumeSpecName "kube-api-access-7pphx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:08.947593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.947566 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls" pod "978a7bc3-9b81-4ee3-b793-4e8963e67567" (UID: "978a7bc3-9b81-4ee3-b793-4e8963e67567"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:08.947593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:08.947582 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs" pod "978a7bc3-9b81-4ee3-b793-4e8963e67567" (UID: "978a7bc3-9b81-4ee3-b793-4e8963e67567"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:09.045467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.042478 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:09.045467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.042510 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pphx\" (UniqueName: \"kubernetes.io/projected/978a7bc3-9b81-4ee3-b793-4e8963e67567-kube-api-access-7pphx\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:09.045467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.042520 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978a7bc3-9b81-4ee3-b793-4e8963e67567-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:09.045467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.042531 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978a7bc3-9b81-4ee3-b793-4e8963e67567-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:09.324736 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.324651 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:09.480203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.480166 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:09.632595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.632497 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:09.743426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.743370 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" Oct 30 21:30:09.743894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.743436 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" event={"ID":"978a7bc3-9b81-4ee3-b793-4e8963e67567","Type":"ContainerDied","Data":"0ac0b2f134cc9677d5392ab5a9bdaeac9fa1abe8a4da8e2e42bb06d93818c31a"} Oct 30 21:30:09.743894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.743498 5039 scope.go:122] "RemoveContainer" containerID="ab16b99ba60d817f7801a4ab97dfd4d8933c165f54bbfeece7359f7357e85c75" Oct 30 21:30:09.792818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.792790 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr"] Oct 30 21:30:09.802573 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.802545 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr"] Oct 30 21:30:09.967554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:09.967523 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:10.763551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:10.763520 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:10.903884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:10.903843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:10.923113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:10.923084 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978a7bc3-9b81-4ee3-b793-4e8963e67567" path="/var/lib/kubelet/pods/978a7bc3-9b81-4ee3-b793-4e8963e67567/volumes" Oct 30 21:30:11.266633 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:11.266590 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:11.747946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:11.747911 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b13793-66bbf657fb-z2ldr" podUID="978a7bc3-9b81-4ee3-b793-4e8963e67567" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.88:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:12.621515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:12.621485 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:12.889627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:12.889593 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:13.741592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:13.741563 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:30:14.510070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:14.510035 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:16.108124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:16.108086 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:16.280037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:16.280004 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:16.455164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:16.455131 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:18.311680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:18.311637 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:18.371531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:18.371502 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:30:19.915172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:19.915135 5039 scope.go:122] "RemoveContainer" containerID="cc0173c3960e948bed5bcacaae99a1f8a97e31bb253ca1aab6381cd3d1af1e0f" Oct 30 21:30:19.915578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:19.915239 5039 scope.go:122] "RemoveContainer" containerID="ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632" Oct 30 21:30:20.504361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:20.504327 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:20.828661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:20.828622 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerStarted","Data":"eb22bfe63377e791c8f2233b17a34c474484f1936d94bc89f4b66ed2b8291db5"} Oct 30 21:30:20.830551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:20.830521 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerStarted","Data":"e64a8d8c8229d30dcd17287f2b2421372f06ee88bb6eedc36acebf2c32ade3a6"} Oct 30 21:30:20.923188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:20.923158 5039 scope.go:122] "RemoveContainer" containerID="51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad" Oct 30 21:30:21.839839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.839807 5039 generic.go:356] "Generic (PLEG): container finished" podID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerID="e64a8d8c8229d30dcd17287f2b2421372f06ee88bb6eedc36acebf2c32ade3a6" exitCode=60 Oct 30 21:30:21.840002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.839866 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerDied","Data":"e64a8d8c8229d30dcd17287f2b2421372f06ee88bb6eedc36acebf2c32ade3a6"} Oct 30 21:30:21.840002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.839961 5039 scope.go:122] "RemoveContainer" containerID="cc0173c3960e948bed5bcacaae99a1f8a97e31bb253ca1aab6381cd3d1af1e0f" Oct 30 21:30:21.840347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.840328 5039 scope.go:122] "RemoveContainer" containerID="e64a8d8c8229d30dcd17287f2b2421372f06ee88bb6eedc36acebf2c32ade3a6" Oct 30 21:30:21.840687 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:21.840665 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc_odh-ai-gateway-infra(619994dc-cf66-4d77-a6bd-7910bc013b63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" Oct 30 21:30:21.848817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.848795 5039 generic.go:356] "Generic (PLEG): container finished" podID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerID="b756c252359e138cefd964d1af1c88cd88e1c096ed25114ca60fc3565449f22c" exitCode=60 Oct 30 21:30:21.848934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.848916 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerDied","Data":"b756c252359e138cefd964d1af1c88cd88e1c096ed25114ca60fc3565449f22c"} Oct 30 21:30:21.849287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.849267 5039 scope.go:122] "RemoveContainer" containerID="b756c252359e138cefd964d1af1c88cd88e1c096ed25114ca60fc3565449f22c" Oct 30 21:30:21.849602 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:21.849576 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx_odh-ai-gateway-infra(f354f2c2-c97a-4083-8ae8-db2b840d9283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" Oct 30 21:30:21.851156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.851138 5039 generic.go:356] "Generic (PLEG): container finished" podID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerID="eb22bfe63377e791c8f2233b17a34c474484f1936d94bc89f4b66ed2b8291db5" exitCode=60 Oct 30 21:30:21.851256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.851235 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerDied","Data":"eb22bfe63377e791c8f2233b17a34c474484f1936d94bc89f4b66ed2b8291db5"} Oct 30 21:30:21.851578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.851555 5039 scope.go:122] "RemoveContainer" containerID="eb22bfe63377e791c8f2233b17a34c474484f1936d94bc89f4b66ed2b8291db5" Oct 30 21:30:21.851818 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:21.851790 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf_odh-ai-gateway-infra(9b430541-0361-4df9-8ce5-be25f8d44e7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" Oct 30 21:30:21.854993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.854971 5039 scope.go:122] "RemoveContainer" containerID="51d367345fc4779da6960be2c09d6ade625644a86f6a2304f11c952a300970ad" Oct 30 21:30:21.867631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.867605 5039 scope.go:122] "RemoveContainer" containerID="ec8d44be3912c36dda1a1b52644d5c01f6d3a188e606efed7d380866e636e632" Oct 30 21:30:21.876188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.876167 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf"] Oct 30 21:30:21.907794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:21.907774 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc"] Oct 30 21:30:22.441247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:22.441215 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:22.928234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:22.927414 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx"] Oct 30 21:30:23.048934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.048903 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:23.107825 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.107788 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp\" (UniqueName: \"kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp\") pod \"9b430541-0361-4df9-8ce5-be25f8d44e7c\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " Oct 30 21:30:23.108038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.108018 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca\") pod \"9b430541-0361-4df9-8ce5-be25f8d44e7c\" (UID: \"9b430541-0361-4df9-8ce5-be25f8d44e7c\") " Oct 30 21:30:23.108576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.108550 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca" pod "9b430541-0361-4df9-8ce5-be25f8d44e7c" (UID: "9b430541-0361-4df9-8ce5-be25f8d44e7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:23.111173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.111110 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp" pod "9b430541-0361-4df9-8ce5-be25f8d44e7c" (UID: "9b430541-0361-4df9-8ce5-be25f8d44e7c"). InnerVolumeSpecName "kube-api-access-47jdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:23.145162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.145140 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:23.148506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.148488 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:23.209273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.209248 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca\") pod \"f354f2c2-c97a-4083-8ae8-db2b840d9283\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " Oct 30 21:30:23.209442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.209339 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz\" (UniqueName: \"kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz\") pod \"f354f2c2-c97a-4083-8ae8-db2b840d9283\" (UID: \"f354f2c2-c97a-4083-8ae8-db2b840d9283\") " Oct 30 21:30:23.209442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.209370 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd\" (UniqueName: \"kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd\") pod \"619994dc-cf66-4d77-a6bd-7910bc013b63\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " Oct 30 21:30:23.209571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.209462 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca\") pod \"619994dc-cf66-4d77-a6bd-7910bc013b63\" (UID: \"619994dc-cf66-4d77-a6bd-7910bc013b63\") " Oct 30 21:30:23.209655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.209628 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca" pod "f354f2c2-c97a-4083-8ae8-db2b840d9283" (UID: "f354f2c2-c97a-4083-8ae8-db2b840d9283"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:23.210118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.210093 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jdp\" (UniqueName: \"kubernetes.io/projected/9b430541-0361-4df9-8ce5-be25f8d44e7c-kube-api-access-47jdp\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.210236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.210116 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca" pod "619994dc-cf66-4d77-a6bd-7910bc013b63" (UID: "619994dc-cf66-4d77-a6bd-7910bc013b63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:23.210236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.210122 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b430541-0361-4df9-8ce5-be25f8d44e7c-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.210236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.210169 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f354f2c2-c97a-4083-8ae8-db2b840d9283-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.211544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.211519 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz" pod "f354f2c2-c97a-4083-8ae8-db2b840d9283" (UID: "f354f2c2-c97a-4083-8ae8-db2b840d9283"). InnerVolumeSpecName "kube-api-access-9b2sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:23.212101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.212077 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd" pod "619994dc-cf66-4d77-a6bd-7910bc013b63" (UID: "619994dc-cf66-4d77-a6bd-7910bc013b63"). InnerVolumeSpecName "kube-api-access-nk5qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:23.240951 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.240925 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:23.311478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.311446 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b2sz\" (UniqueName: \"kubernetes.io/projected/f354f2c2-c97a-4083-8ae8-db2b840d9283-kube-api-access-9b2sz\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.311478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.311473 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5qd\" (UniqueName: \"kubernetes.io/projected/619994dc-cf66-4d77-a6bd-7910bc013b63-kube-api-access-nk5qd\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.311627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.311488 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619994dc-cf66-4d77-a6bd-7910bc013b63-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.871983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.871948 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" Oct 30 21:30:23.871983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.871968 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc" event={"ID":"619994dc-cf66-4d77-a6bd-7910bc013b63","Type":"ContainerDied","Data":"e028988b71da81cc42217f818e9e943ccda410464f3959288004c309bb27745b"} Oct 30 21:30:23.872498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.872028 5039 scope.go:122] "RemoveContainer" containerID="e64a8d8c8229d30dcd17287f2b2421372f06ee88bb6eedc36acebf2c32ade3a6" Oct 30 21:30:23.873500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.873483 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" Oct 30 21:30:23.873627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.873500 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx" event={"ID":"f354f2c2-c97a-4083-8ae8-db2b840d9283","Type":"ContainerDied","Data":"110c192e51471e29bbcaa6a0f5c516d6948421db6f00d4963829dcd4d8faee1d"} Oct 30 21:30:23.875208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.875179 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" Oct 30 21:30:23.875336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.875210 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf" event={"ID":"9b430541-0361-4df9-8ce5-be25f8d44e7c","Type":"ContainerDied","Data":"7b101a503eb8fbe8180f0afb7b278c97b8270621fb8c55d75801b4b17fa668fd"} Oct 30 21:30:23.895931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.895885 5039 scope.go:122] "RemoveContainer" containerID="b756c252359e138cefd964d1af1c88cd88e1c096ed25114ca60fc3565449f22c" Oct 30 21:30:23.906110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.906093 5039 scope.go:122] "RemoveContainer" containerID="eb22bfe63377e791c8f2233b17a34c474484f1936d94bc89f4b66ed2b8291db5" Oct 30 21:30:23.952939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.952915 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf"] Oct 30 21:30:23.962693 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.962673 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889930-wl5hf"] Oct 30 21:30:23.986836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.986815 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc"] Oct 30 21:30:23.990745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:23.990721 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3f2f4ebe-29889930-rmgrc"] Oct 30 21:30:24.019447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:24.019427 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx"] Oct 30 21:30:24.024584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:24.024565 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889930-bn9dx"] Oct 30 21:30:24.308903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:24.308853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:24.922158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:24.922126 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" path="/var/lib/kubelet/pods/619994dc-cf66-4d77-a6bd-7910bc013b63/volumes" Oct 30 21:30:24.922646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:24.922536 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" path="/var/lib/kubelet/pods/9b430541-0361-4df9-8ce5-be25f8d44e7c/volumes" Oct 30 21:30:24.922896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:24.922882 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" path="/var/lib/kubelet/pods/f354f2c2-c97a-4083-8ae8-db2b840d9283/volumes" Oct 30 21:30:25.184234 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:25.184158 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:30:26.275364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:26.275309 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:27.494597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:27.494556 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:27.636173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:27.636122 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:28.011805 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:28.011768 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:28.510067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:28.510023 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:29.593877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:29.593843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:29.880261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:29.880180 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:31.833051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:31.833015 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:33.812815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:33.812778 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:35.527592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:35.527553 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:37.154489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:37.154440 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:38.796603 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:38.795237 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:38.886522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:38.886477 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj"] Oct 30 21:30:38.925555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:38.925494 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97"] Oct 30 21:30:39.080489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:39.080380 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:39.228406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:39.228357 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:40.384761 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:40.384730 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:40.533421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:40.533350 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:42.118917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:42.118881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.691733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:43.691671 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.887628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:43.887589 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" podUID="251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" containerName="payload-processing" containerID="cri-o://5f346d1818a66757aba31a8e0dbf9768bad2eea8b4d45ab3e625c428349fcf56" gracePeriod=25 Oct 30 21:30:43.925934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:43.925899 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" podUID="22e77cc5-aa29-4154-ba46-f476d8c9d1fe" containerName="payload-pre-processing" containerID="cri-o://ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9" gracePeriod=25 Oct 30 21:30:44.021989 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.021960 5039 generic.go:356] "Generic (PLEG): container finished" podID="251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" containerID="5f346d1818a66757aba31a8e0dbf9768bad2eea8b4d45ab3e625c428349fcf56" exitCode=0 Oct 30 21:30:44.022110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.022029 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" event={"ID":"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e","Type":"ContainerDied","Data":"5f346d1818a66757aba31a8e0dbf9768bad2eea8b4d45ab3e625c428349fcf56"} Oct 30 21:30:44.211692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.211631 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:30:44.242106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.242081 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:30:44.289360 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.289327 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l\" (UniqueName: \"kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l\") pod \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " Oct 30 21:30:44.289557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.289432 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume\") pod \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\" (UID: \"22e77cc5-aa29-4154-ba46-f476d8c9d1fe\") " Oct 30 21:30:44.289557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.289503 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h\" (UniqueName: \"kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h\") pod \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " Oct 30 21:30:44.289717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.289672 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume\") pod \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\" (UID: \"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e\") " Oct 30 21:30:44.290185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.290155 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume" pod "251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" (UID: "251c5c30-0b6b-4d86-a3f7-2ca7846ce65e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:44.291203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.291160 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume" pod "22e77cc5-aa29-4154-ba46-f476d8c9d1fe" (UID: "22e77cc5-aa29-4154-ba46-f476d8c9d1fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:44.293726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.293697 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l" pod "22e77cc5-aa29-4154-ba46-f476d8c9d1fe" (UID: "22e77cc5-aa29-4154-ba46-f476d8c9d1fe"). InnerVolumeSpecName "kube-api-access-nkk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:44.293878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.293857 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h" pod "251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" (UID: "251c5c30-0b6b-4d86-a3f7-2ca7846ce65e"). InnerVolumeSpecName "kube-api-access-c9n2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:44.391404 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.391369 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:44.391546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.391425 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkk5l\" (UniqueName: \"kubernetes.io/projected/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-kube-api-access-nkk5l\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:44.391546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.391437 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e77cc5-aa29-4154-ba46-f476d8c9d1fe-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:44.391546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:44.391445 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9n2h\" (UniqueName: \"kubernetes.io/projected/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e-kube-api-access-c9n2h\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:45.029200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.029169 5039 generic.go:356] "Generic (PLEG): container finished" podID="22e77cc5-aa29-4154-ba46-f476d8c9d1fe" containerID="ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9" exitCode=0 Oct 30 21:30:45.029618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.029218 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" Oct 30 21:30:45.029618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.029255 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" event={"ID":"22e77cc5-aa29-4154-ba46-f476d8c9d1fe","Type":"ContainerDied","Data":"ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9"} Oct 30 21:30:45.029618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.029304 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97" event={"ID":"22e77cc5-aa29-4154-ba46-f476d8c9d1fe","Type":"ContainerDied","Data":"2ac49b13e59116001d8d51f0c53253077c4dd0738ce9bceb90e17a65947b354c"} Oct 30 21:30:45.029618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.029326 5039 scope.go:122] "RemoveContainer" containerID="ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9" Oct 30 21:30:45.031058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.031032 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" event={"ID":"251c5c30-0b6b-4d86-a3f7-2ca7846ce65e","Type":"ContainerDied","Data":"0f3e0d3cd18e4bf11cdb67eb55b42f49de5273f9c96ddd20b0bfc5b79fd2696e"} Oct 30 21:30:45.031058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.031052 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj" Oct 30 21:30:45.039174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.039151 5039 scope.go:122] "RemoveContainer" containerID="ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9" Oct 30 21:30:45.039447 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:45.039429 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9\": container with ID starting with ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9 not found: ID does not exist" containerID="ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9" Oct 30 21:30:45.039520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.039454 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9"} err="failed to get container status \"ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9\": rpc error: code = NotFound desc = could not find container \"ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9\": container with ID starting with ab8d76ff69ea90b92b1a0f39f50e0aa125ea1b54f1e984e048346840849822e9 not found: ID does not exist" Oct 30 21:30:45.039520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.039471 5039 scope.go:122] "RemoveContainer" containerID="5f346d1818a66757aba31a8e0dbf9768bad2eea8b4d45ab3e625c428349fcf56" Oct 30 21:30:45.076136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.076113 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj"] Oct 30 21:30:45.084108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.084084 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3f2f4ebe-7bd78774d9-rj4lj"] Oct 30 21:30:45.119235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.119207 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97"] Oct 30 21:30:45.120986 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.120967 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-2lg97"] Oct 30 21:30:45.348892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:45.348174 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:46.922207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:46.922177 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e77cc5-aa29-4154-ba46-f476d8c9d1fe" path="/var/lib/kubelet/pods/22e77cc5-aa29-4154-ba46-f476d8c9d1fe/volumes" Oct 30 21:30:46.922581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:46.922565 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" path="/var/lib/kubelet/pods/251c5c30-0b6b-4d86-a3f7-2ca7846ce65e/volumes" Oct 30 21:30:47.088437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:47.088404 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:48.447143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.447104 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql"] Oct 30 21:30:48.447679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.447372 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" podUID="57bfdae8-8de2-4f34-8f77-9b3f550f458e" containerName="maas-api" containerID="cri-o://c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5" gracePeriod=30 Oct 30 21:30:48.560072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.559504 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:48.819845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.819813 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:30:48.948576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.948543 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs\") pod \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " Oct 30 21:30:48.948792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.948775 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476\" (UniqueName: \"kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476\") pod \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " Oct 30 21:30:48.948868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.948856 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca\") pod \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " Oct 30 21:30:48.948980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.948923 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\") pod \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\" (UID: \"57bfdae8-8de2-4f34-8f77-9b3f550f458e\") " Oct 30 21:30:48.949260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.949227 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca" pod "57bfdae8-8de2-4f34-8f77-9b3f550f458e" (UID: "57bfdae8-8de2-4f34-8f77-9b3f550f458e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:48.949686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.949660 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bfdae8-8de2-4f34-8f77-9b3f550f458e-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:48.951019 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.950992 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs" pod "57bfdae8-8de2-4f34-8f77-9b3f550f458e" (UID: "57bfdae8-8de2-4f34-8f77-9b3f550f458e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:48.951259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.951233 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476" pod "57bfdae8-8de2-4f34-8f77-9b3f550f458e" (UID: "57bfdae8-8de2-4f34-8f77-9b3f550f458e"). InnerVolumeSpecName "kube-api-access-65476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:48.951586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:48.951559 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls" pod "57bfdae8-8de2-4f34-8f77-9b3f550f458e" (UID: "57bfdae8-8de2-4f34-8f77-9b3f550f458e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:49.051868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.051762 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65476\" (UniqueName: \"kubernetes.io/projected/57bfdae8-8de2-4f34-8f77-9b3f550f458e-kube-api-access-65476\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:49.051868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.051812 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:49.051868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.051827 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bfdae8-8de2-4f34-8f77-9b3f550f458e-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:30:49.066614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.066569 5039 generic.go:356] "Generic (PLEG): container finished" podID="57bfdae8-8de2-4f34-8f77-9b3f550f458e" containerID="c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5" exitCode=0 Oct 30 21:30:49.066808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.066783 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" event={"ID":"57bfdae8-8de2-4f34-8f77-9b3f550f458e","Type":"ContainerDied","Data":"c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5"} Oct 30 21:30:49.066879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.066834 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" event={"ID":"57bfdae8-8de2-4f34-8f77-9b3f550f458e","Type":"ContainerDied","Data":"6521443ef7e04ed06df05a5d15843f9a6167d8b3907e3cbcf1ba4e6e97d908ae"} Oct 30 21:30:49.066879 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.066868 5039 scope.go:122] "RemoveContainer" containerID="c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5" Oct 30 21:30:49.067055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.067038 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql" Oct 30 21:30:49.123359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.123206 5039 scope.go:122] "RemoveContainer" containerID="c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5" Oct 30 21:30:49.124382 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:49.124349 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5\": container with ID starting with c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5 not found: ID does not exist" containerID="c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5" Oct 30 21:30:49.124572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.124539 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5"} err="failed to get container status \"c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5\": rpc error: code = NotFound desc = could not find container \"c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5\": container with ID starting with c1fb376c8fd493dab56caea732a95c8dd074424f5263f324f2a101a74dbd7ee5 not found: ID does not exist" Oct 30 21:30:49.139225 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.139188 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql"] Oct 30 21:30:49.149739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.149712 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3f2f4ebe-c779dff59-jxjql"] Oct 30 21:30:49.254922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.254894 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:49.408757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.408715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:49.555649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.555615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:49.881441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:49.881322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:50.922365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:50.922334 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bfdae8-8de2-4f34-8f77-9b3f550f458e" path="/var/lib/kubelet/pods/57bfdae8-8de2-4f34-8f77-9b3f550f458e/volumes" Oct 30 21:30:51.219924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:51.219801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:52.901609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:52.901576 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:54.605460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:54.605417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:54.797400 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:54.797352 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd"] Oct 30 21:30:54.914203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:54.914155 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:55.185771 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:30:55.185698 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:30:56.293730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:56.293696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:57.907210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:57.907172 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:58.340344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:58.340260 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:30:59.811442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:30:59.811407 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.301016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:01.300978 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.734415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:01.734036 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.892115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:01.892078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.072840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:02.072743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.636972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:02.636897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.791367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:02.791320 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:03.235495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:03.235460 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:04.492611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:04.492576 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:06.270836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:06.270801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:07.266329 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:07.266277 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:07.748883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:07.748851 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:07.932409 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:07.932243 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:08.118385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:08.118294 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:09.357523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.357485 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:09.510819 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.510781 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:09.635927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.635889 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt"] Oct 30 21:31:09.636229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.636182 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" containerID="cri-o://d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab" gracePeriod=30 Oct 30 21:31:09.704492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.704459 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:09.818621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.818576 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" podUID="a6be7076-59d7-4cf4-96a0-eb9edb608afb" containerName="main" containerID="cri-o://8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428" gracePeriod=45 Oct 30 21:31:09.876362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:09.876225 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:10.124595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.124570 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:31:10.167281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.167254 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4"] Oct 30 21:31:10.167496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.167473 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" containerID="cri-o://298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119" gracePeriod=30 Oct 30 21:31:10.225785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.225697 5039 generic.go:356] "Generic (PLEG): container finished" podID="a6be7076-59d7-4cf4-96a0-eb9edb608afb" containerID="8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428" exitCode=0 Oct 30 21:31:10.225785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.225734 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" Oct 30 21:31:10.225785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.225768 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" event={"ID":"a6be7076-59d7-4cf4-96a0-eb9edb608afb","Type":"ContainerDied","Data":"8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428"} Oct 30 21:31:10.226066 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.225811 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd" event={"ID":"a6be7076-59d7-4cf4-96a0-eb9edb608afb","Type":"ContainerDied","Data":"bf15d16f74d005989033b6f9a14dcb5f1f0f9f308a1345fda21f19123b968961"} Oct 30 21:31:10.226066 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.225833 5039 scope.go:122] "RemoveContainer" containerID="8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428" Oct 30 21:31:10.237077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.237053 5039 scope.go:122] "RemoveContainer" containerID="8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428" Oct 30 21:31:10.237410 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:10.237372 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428\": container with ID starting with 8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428 not found: ID does not exist" containerID="8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428" Oct 30 21:31:10.237501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.237424 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428"} err="failed to get container status \"8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428\": rpc error: code = NotFound desc = could not find container \"8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428\": container with ID starting with 8cdb54a96ccb44cdd7e1ebcc46c44b3b9f7f58a3ee0c9279d4ba0a8559bbf428 not found: ID does not exist" Oct 30 21:31:10.262432 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262378 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache\") pod \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " Oct 30 21:31:10.262542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262477 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp\" (UniqueName: \"kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp\") pod \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " Oct 30 21:31:10.262608 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262585 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm\") pod \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " Oct 30 21:31:10.262670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262628 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir\") pod \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " Oct 30 21:31:10.262670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262625 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache" pod "a6be7076-59d7-4cf4-96a0-eb9edb608afb" (UID: "a6be7076-59d7-4cf4-96a0-eb9edb608afb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:10.262670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262661 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home\") pod \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " Oct 30 21:31:10.262827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262809 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs\") pod \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\" (UID: \"a6be7076-59d7-4cf4-96a0-eb9edb608afb\") " Oct 30 21:31:10.262895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262870 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir" pod "a6be7076-59d7-4cf4-96a0-eb9edb608afb" (UID: "a6be7076-59d7-4cf4-96a0-eb9edb608afb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:10.262988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.262966 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home" pod "a6be7076-59d7-4cf4-96a0-eb9edb608afb" (UID: "a6be7076-59d7-4cf4-96a0-eb9edb608afb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:10.263312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.263287 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:10.263312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.263309 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:10.263477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.263318 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:10.264645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.264625 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm" pod "a6be7076-59d7-4cf4-96a0-eb9edb608afb" (UID: "a6be7076-59d7-4cf4-96a0-eb9edb608afb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:10.264719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.264655 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs" pod "a6be7076-59d7-4cf4-96a0-eb9edb608afb" (UID: "a6be7076-59d7-4cf4-96a0-eb9edb608afb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:10.264719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.264670 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp" pod "a6be7076-59d7-4cf4-96a0-eb9edb608afb" (UID: "a6be7076-59d7-4cf4-96a0-eb9edb608afb"). InnerVolumeSpecName "kube-api-access-msfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:10.329681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.329646 5039 patch_prober.go:29] interesting pod/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:31:10.329834 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.329701 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:31:10.364548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.364517 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6be7076-59d7-4cf4-96a0-eb9edb608afb-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:10.364548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.364544 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6be7076-59d7-4cf4-96a0-eb9edb608afb-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:10.364929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.364555 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfsp\" (UniqueName: \"kubernetes.io/projected/a6be7076-59d7-4cf4-96a0-eb9edb608afb-kube-api-access-msfsp\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:10.567213 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.567181 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd"] Oct 30 21:31:10.571562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.571537 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3f2f4ebe/e2e-lifecycle-model-3f2f4ebe-kserve-68b4867b89-65tgd"] Oct 30 21:31:10.588527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.588501 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:10.746904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.746867 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:10.923857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:10.923820 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6be7076-59d7-4cf4-96a0-eb9edb608afb" path="/var/lib/kubelet/pods/a6be7076-59d7-4cf4-96a0-eb9edb608afb/volumes" Oct 30 21:31:11.044472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:11.044436 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:11.381357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:11.380855 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:11.558647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:11.558601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:12.052092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:12.051094 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:12.743105 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:12.743071 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:13.220181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.220149 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:13.383737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.383700 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:13.450108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.450075 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j"] Oct 30 21:31:13.451662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451634 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.451662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451661 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451687 5039 cpu_manager.go:414] "containerMap: removing container" podUID="a6be7076-59d7-4cf4-96a0-eb9edb608afb" containerName="main" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451695 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6be7076-59d7-4cf4-96a0-eb9edb608afb" containerName="main" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451718 5039 cpu_manager.go:414] "containerMap: removing container" podUID="22e77cc5-aa29-4154-ba46-f476d8c9d1fe" containerName="payload-pre-processing" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451725 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e77cc5-aa29-4154-ba46-f476d8c9d1fe" containerName="payload-pre-processing" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451748 5039 cpu_manager.go:414] "containerMap: removing container" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451756 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451775 5039 cpu_manager.go:414] "containerMap: removing container" podUID="978a7bc3-9b81-4ee3-b793-4e8963e67567" containerName="maas-api" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451781 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978a7bc3-9b81-4ee3-b793-4e8963e67567" containerName="maas-api" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451791 5039 cpu_manager.go:414] "containerMap: removing container" podUID="57bfdae8-8de2-4f34-8f77-9b3f550f458e" containerName="maas-api" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451803 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bfdae8-8de2-4f34-8f77-9b3f550f458e" containerName="maas-api" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451816 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451823 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451844 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451852 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451863 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.451864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451870 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451885 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451892 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451904 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451911 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451920 5039 cpu_manager.go:414] "containerMap: removing container" podUID="251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" containerName="payload-processing" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451929 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" containerName="payload-processing" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451939 5039 cpu_manager.go:414] "containerMap: removing container" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.451945 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452099 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452110 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452118 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452126 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="251c5c30-0b6b-4d86-a3f7-2ca7846ce65e" containerName="payload-processing" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452135 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452148 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452158 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6be7076-59d7-4cf4-96a0-eb9edb608afb" containerName="main" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452168 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e77cc5-aa29-4154-ba46-f476d8c9d1fe" containerName="payload-pre-processing" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452182 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="978a7bc3-9b81-4ee3-b793-4e8963e67567" containerName="maas-api" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452190 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bfdae8-8de2-4f34-8f77-9b3f550f458e" containerName="maas-api" Oct 30 21:31:13.452708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.452199 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:13.457806 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.457780 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.502069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.500674 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-229795\"" Oct 30 21:31:13.502069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.500855 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-229795-dockercfg-cx74b\"" Oct 30 21:31:13.504319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.504295 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j"] Oct 30 21:31:13.612206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.612168 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.612440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.612252 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5\") pod \"payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.714323 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.714237 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.714644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.714352 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5\") pod \"payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.715545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.715518 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.731727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.731705 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5\") pod \"payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:13.806714 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:13.806651 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:14.102601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.102573 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j"] Oct 30 21:31:14.103803 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:31:14.103774 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d1a9bd_497f_49c2_879e_a085491c8ea1.slice/crio-ecf84ca27ecf51351d2bb025a7fd0f48d1bc56d796f07785190fc0c12e8553b7 WatchSource:0}: Error finding container ecf84ca27ecf51351d2bb025a7fd0f48d1bc56d796f07785190fc0c12e8553b7: Status 404 returned error can't find the container with id ecf84ca27ecf51351d2bb025a7fd0f48d1bc56d796f07785190fc0c12e8553b7 Oct 30 21:31:14.222846 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.222814 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:14.263989 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.263947 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" event={"ID":"81d1a9bd-497f-49c2-879e-a085491c8ea1","Type":"ContainerStarted","Data":"0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f"} Oct 30 21:31:14.264156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.264018 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:14.264156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.264046 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" event={"ID":"81d1a9bd-497f-49c2-879e-a085491c8ea1","Type":"ContainerStarted","Data":"ecf84ca27ecf51351d2bb025a7fd0f48d1bc56d796f07785190fc0c12e8553b7"} Oct 30 21:31:14.299045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.298520 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" podStartSLOduration=1.298499714 podStartE2EDuration="1.298499714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:14.298180586 +0000 UTC m=+1313.508120216" watchObservedRunningTime="2026-10-30 21:31:14.298499714 +0000 UTC m=+1313.508439344" Oct 30 21:31:14.928522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:14.928496 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:31:15.030754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.030713 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031359 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031422 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031483 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031511 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031594 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031646 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031679 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.031893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.031828 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data\") pod \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\" (UID: \"11e1cd19-38e4-4325-9ac4-1294eca4a5c4\") " Oct 30 21:31:15.032443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.032409 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.033014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.032984 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.033464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.033438 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.033892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.033865 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.034154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.034129 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:15.036407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.036366 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:15.036527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.036507 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "kube-api-access-mrwr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:15.036752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.036728 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.036824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.036753 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo" pod "11e1cd19-38e4-4325-9ac4-1294eca4a5c4" (UID: "11e1cd19-38e4-4325-9ac4-1294eca4a5c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:15.037841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037814 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.037908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037846 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.037908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037860 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.037908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037873 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.037908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037885 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.037908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037898 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwr7\" (UniqueName: \"kubernetes.io/projected/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-kube-api-access-mrwr7\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.038107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037912 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.038107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037928 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.038107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.037942 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1cd19-38e4-4325-9ac4-1294eca4a5c4-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.277048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.277015 5039 generic.go:356] "Generic (PLEG): container finished" podID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerID="d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab" exitCode=0 Oct 30 21:31:15.277216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.277072 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" Oct 30 21:31:15.277216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.277100 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" event={"ID":"11e1cd19-38e4-4325-9ac4-1294eca4a5c4","Type":"ContainerDied","Data":"d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab"} Oct 30 21:31:15.277216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.277160 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt" event={"ID":"11e1cd19-38e4-4325-9ac4-1294eca4a5c4","Type":"ContainerDied","Data":"690e6df1895e0aaac5cfadddf08d01a878a86f9b774a718ade9ca4997efe78cc"} Oct 30 21:31:15.277216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.277187 5039 scope.go:122] "RemoveContainer" containerID="d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab" Oct 30 21:31:15.287173 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.287159 5039 scope.go:122] "RemoveContainer" containerID="134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97" Oct 30 21:31:15.293746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.293720 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:15.296801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.296773 5039 scope.go:122] "RemoveContainer" containerID="d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab" Oct 30 21:31:15.297085 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:15.297063 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab\": container with ID starting with d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab not found: ID does not exist" containerID="d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab" Oct 30 21:31:15.297169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.297093 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab"} err="failed to get container status \"d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab\": rpc error: code = NotFound desc = could not find container \"d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab\": container with ID starting with d942841c7db1d4031a52262fb02db27087d27d45f565acb3efbe4410c5baeaab not found: ID does not exist" Oct 30 21:31:15.297169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.297109 5039 scope.go:122] "RemoveContainer" containerID="134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97" Oct 30 21:31:15.297426 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:15.297376 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97\": container with ID starting with 134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97 not found: ID does not exist" containerID="134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97" Oct 30 21:31:15.297515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.297433 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97"} err="failed to get container status \"134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97\": rpc error: code = NotFound desc = could not find container \"134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97\": container with ID starting with 134c3bd2c715853b60ec355e79f12214bfe961a0f32486ed70dc410177d9fd97 not found: ID does not exist" Oct 30 21:31:15.323262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.323235 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt"] Oct 30 21:31:15.332639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.332616 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c6dab8-openshift-default-686857879f-7xwwt"] Oct 30 21:31:15.429977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.429921 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:31:15.461649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.461619 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:15.543020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.542987 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543072 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543122 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543158 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543195 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543271 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543443 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.543570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543472 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543508 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.543570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543543 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.543703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543606 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.543703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543611 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543658 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7\") pod \"01841fb2-c4bb-4577-96b1-81b3756b0105\" (UID: \"01841fb2-c4bb-4577-96b1-81b3756b0105\") " Oct 30 21:31:15.543973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.543946 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:15.544363 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.544341 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.544486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.544368 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.544486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.544384 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.544486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.544411 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.544486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.544425 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01841fb2-c4bb-4577-96b1-81b3756b0105-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.545542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.545517 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.545631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.545524 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:15.545841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.545815 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:15.545920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.545823 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7" pod "01841fb2-c4bb-4577-96b1-81b3756b0105" (UID: "01841fb2-c4bb-4577-96b1-81b3756b0105"). InnerVolumeSpecName "kube-api-access-xwhf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:15.646026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.645999 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.646026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.646026 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwhf7\" (UniqueName: \"kubernetes.io/projected/01841fb2-c4bb-4577-96b1-81b3756b0105-kube-api-access-xwhf7\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.646190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.646036 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:15.646190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:15.646045 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01841fb2-c4bb-4577-96b1-81b3756b0105-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:16.227076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.227036 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:16.286090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.286053 5039 generic.go:356] "Generic (PLEG): container finished" podID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerID="298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119" exitCode=0 Oct 30 21:31:16.286259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.286143 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" event={"ID":"01841fb2-c4bb-4577-96b1-81b3756b0105","Type":"ContainerDied","Data":"298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119"} Oct 30 21:31:16.286259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.286155 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" Oct 30 21:31:16.286259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.286256 5039 scope.go:122] "RemoveContainer" containerID="298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119" Oct 30 21:31:16.286503 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.286243 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4" event={"ID":"01841fb2-c4bb-4577-96b1-81b3756b0105","Type":"ContainerDied","Data":"f17f82f62143b5d42c1b6f3caefe6002761a155052ec3c1bbb66c2e6fa2cecd5"} Oct 30 21:31:16.296413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.296374 5039 scope.go:122] "RemoveContainer" containerID="298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119" Oct 30 21:31:16.296667 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:16.296650 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119\": container with ID starting with 298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119 not found: ID does not exist" containerID="298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119" Oct 30 21:31:16.296738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.296674 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119"} err="failed to get container status \"298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119\": rpc error: code = NotFound desc = could not find container \"298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119\": container with ID starting with 298b9693f8e582dbcb1c09a82e5c940c13b535c8eab2a7d6bba1cdf83756b119 not found: ID does not exist" Oct 30 21:31:16.332526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.332495 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4"] Oct 30 21:31:16.341494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.341471 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-n2rn4"] Oct 30 21:31:16.922122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.922094 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" path="/var/lib/kubelet/pods/01841fb2-c4bb-4577-96b1-81b3756b0105/volumes" Oct 30 21:31:16.922597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:16.922583 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" path="/var/lib/kubelet/pods/11e1cd19-38e4-4325-9ac4-1294eca4a5c4/volumes" Oct 30 21:31:17.105139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:17.105109 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:17.282931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:17.282836 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:18.295966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:18.295928 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:19.992935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:19.992900 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:21.620348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:21.620252 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:22.148475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:22.148444 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542"] Oct 30 21:31:22.183962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:22.183931 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j"] Oct 30 21:31:22.337213 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:22.337160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:22.483983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:22.483892 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:23.386100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:23.386067 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:24.906481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:24.906432 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:25.050995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:25.050934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:25.187545 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:25.187459 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:31:25.207301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:25.207265 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:25.281579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:25.281558 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:26.711822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:26.711769 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:27.149524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.149495 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" podUID="560f36b7-e525-4c67-b0b8-3b6bc794bcc4" containerName="payload-processing" containerID="cri-o://03ecbc383005df1fe972c1de43ea2cca50a0c33704f4e3f4a97e921db1c291a0" gracePeriod=25 Oct 30 21:31:27.185090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.185062 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" podUID="1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" containerName="payload-pre-processing" containerID="cri-o://f44701ff9f6c1141d93ba759b79cb1d5b3525518fd5b38b1e561f179e459474a" gracePeriod=25 Oct 30 21:31:27.369532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.369497 5039 generic.go:356] "Generic (PLEG): container finished" podID="560f36b7-e525-4c67-b0b8-3b6bc794bcc4" containerID="03ecbc383005df1fe972c1de43ea2cca50a0c33704f4e3f4a97e921db1c291a0" exitCode=0 Oct 30 21:31:27.369787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.369757 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" event={"ID":"560f36b7-e525-4c67-b0b8-3b6bc794bcc4","Type":"ContainerDied","Data":"03ecbc383005df1fe972c1de43ea2cca50a0c33704f4e3f4a97e921db1c291a0"} Oct 30 21:31:27.372254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.371979 5039 generic.go:356] "Generic (PLEG): container finished" podID="1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" containerID="f44701ff9f6c1141d93ba759b79cb1d5b3525518fd5b38b1e561f179e459474a" exitCode=0 Oct 30 21:31:27.372254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.372190 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" event={"ID":"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d","Type":"ContainerDied","Data":"f44701ff9f6c1141d93ba759b79cb1d5b3525518fd5b38b1e561f179e459474a"} Oct 30 21:31:27.494203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.494178 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:31:27.496430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.496411 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:31:27.614298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614270 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume\") pod \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " Oct 30 21:31:27.614500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614308 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj\" (UniqueName: \"kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj\") pod \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\" (UID: \"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d\") " Oct 30 21:31:27.614500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614445 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume\") pod \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " Oct 30 21:31:27.614500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614471 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh\" (UniqueName: \"kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh\") pod \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\" (UID: \"560f36b7-e525-4c67-b0b8-3b6bc794bcc4\") " Oct 30 21:31:27.614648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614630 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume" pod "1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" (UID: "1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:27.614831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614808 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume" pod "560f36b7-e525-4c67-b0b8-3b6bc794bcc4" (UID: "560f36b7-e525-4c67-b0b8-3b6bc794bcc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:27.614944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614929 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:27.614987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.614947 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:27.616553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.616529 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj" pod "1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" (UID: "1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d"). InnerVolumeSpecName "kube-api-access-q78lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:27.616553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.616542 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh" pod "560f36b7-e525-4c67-b0b8-3b6bc794bcc4" (UID: "560f36b7-e525-4c67-b0b8-3b6bc794bcc4"). InnerVolumeSpecName "kube-api-access-tlsfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:27.715683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.715623 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlsfh\" (UniqueName: \"kubernetes.io/projected/560f36b7-e525-4c67-b0b8-3b6bc794bcc4-kube-api-access-tlsfh\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:27.715683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:27.715648 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q78lj\" (UniqueName: \"kubernetes.io/projected/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d-kube-api-access-q78lj\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:28.323488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.323453 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt"] Oct 30 21:31:28.324032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324014 5039 cpu_manager.go:414] "containerMap: removing container" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:28.324121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324032 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:28.324121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324050 5039 cpu_manager.go:414] "containerMap: removing container" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" Oct 30 21:31:28.324121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324057 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" Oct 30 21:31:28.324121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324096 5039 cpu_manager.go:414] "containerMap: removing container" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" Oct 30 21:31:28.324121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324105 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324130 5039 cpu_manager.go:414] "containerMap: removing container" podUID="560f36b7-e525-4c67-b0b8-3b6bc794bcc4" containerName="payload-processing" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324137 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560f36b7-e525-4c67-b0b8-3b6bc794bcc4" containerName="payload-processing" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324151 5039 cpu_manager.go:414] "containerMap: removing container" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324157 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324168 5039 cpu_manager.go:414] "containerMap: removing container" podUID="1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" containerName="payload-pre-processing" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324174 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" containerName="payload-pre-processing" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324331 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="560f36b7-e525-4c67-b0b8-3b6bc794bcc4" containerName="payload-processing" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324346 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="01841fb2-c4bb-4577-96b1-81b3756b0105" containerName="istio-proxy" Oct 30 21:31:28.324366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324353 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" containerName="payload-pre-processing" Oct 30 21:31:28.324790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324370 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f354f2c2-c97a-4083-8ae8-db2b840d9283" containerName="cleanup" Oct 30 21:31:28.324790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324378 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b430541-0361-4df9-8ce5-be25f8d44e7c" containerName="cleanup" Oct 30 21:31:28.324790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324405 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="619994dc-cf66-4d77-a6bd-7910bc013b63" containerName="cleanup" Oct 30 21:31:28.324790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324416 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" Oct 30 21:31:28.324790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.324429 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1cd19-38e4-4325-9ac4-1294eca4a5c4" containerName="istio-proxy" Oct 30 21:31:28.328143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.328124 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.364449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.357293 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c027ae-serving-cert\"" Oct 30 21:31:28.364449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.362404 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt"] Oct 30 21:31:28.379922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.379877 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" Oct 30 21:31:28.381529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.379910 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542" event={"ID":"560f36b7-e525-4c67-b0b8-3b6bc794bcc4","Type":"ContainerDied","Data":"1fc368277a27019b43ecde66903e3593803d9a9a97dfadde3725e4780981c878"} Oct 30 21:31:28.381529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.379975 5039 scope.go:122] "RemoveContainer" containerID="03ecbc383005df1fe972c1de43ea2cca50a0c33704f4e3f4a97e921db1c291a0" Oct 30 21:31:28.383762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.383735 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" event={"ID":"1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d","Type":"ContainerDied","Data":"22aa91caa31892ec0efcf8a1dba9a45c54e76971b0382d67863db4dc9430b5a8"} Oct 30 21:31:28.383922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.383847 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j" Oct 30 21:31:28.394753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.394645 5039 scope.go:122] "RemoveContainer" containerID="f44701ff9f6c1141d93ba759b79cb1d5b3525518fd5b38b1e561f179e459474a" Oct 30 21:31:28.421428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.421368 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.421543 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.421525 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7br8b\" (UniqueName: \"kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.421650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.421634 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.421701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.421688 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.440743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.440720 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542"] Oct 30 21:31:28.446169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.446147 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-342fc9-674dbd9c95-m6542"] Oct 30 21:31:28.460719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.460693 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:28.474946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.474927 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j"] Oct 30 21:31:28.479602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.479580 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-342fc9-7bd99f8ddc-5tj2j"] Oct 30 21:31:28.522473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.522448 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.522570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.522517 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.522657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.522643 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.522758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.522744 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7br8b\" (UniqueName: \"kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.523197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.523177 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.524826 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.524810 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.524883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.524872 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.539755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.539735 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7br8b\" (UniqueName: \"kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b\") pod \"maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.611229 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.611188 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:28.640518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.640490 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:28.922799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.922764 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d" path="/var/lib/kubelet/pods/1e8a7702-1a82-43e8-8f0c-d4bcaa110e5d/volumes" Oct 30 21:31:28.923335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.923316 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="560f36b7-e525-4c67-b0b8-3b6bc794bcc4" path="/var/lib/kubelet/pods/560f36b7-e525-4c67-b0b8-3b6bc794bcc4/volumes" Oct 30 21:31:28.957481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:28.957457 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt"] Oct 30 21:31:28.958745 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:31:28.958716 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dcdf53_89b6_4302_94ef_d59d05d1f718.slice/crio-9beeb6792dc750d41b92ed0086a51884da75a8464313c9bc08923195906c6df2 WatchSource:0}: Error finding container 9beeb6792dc750d41b92ed0086a51884da75a8464313c9bc08923195906c6df2: Status 404 returned error can't find the container with id 9beeb6792dc750d41b92ed0086a51884da75a8464313c9bc08923195906c6df2 Oct 30 21:31:29.393139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:29.393106 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" event={"ID":"06dcdf53-89b6-4302-94ef-d59d05d1f718","Type":"ContainerStarted","Data":"14fa04e59f63586bac833682d79382912da37de752b41f42c108c39c006efba9"} Oct 30 21:31:29.393295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:29.393204 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" event={"ID":"06dcdf53-89b6-4302-94ef-d59d05d1f718","Type":"ContainerStarted","Data":"9beeb6792dc750d41b92ed0086a51884da75a8464313c9bc08923195906c6df2"} Oct 30 21:31:29.393438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:29.393416 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:29.431099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:29.431035 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" podStartSLOduration=1.124185212 podStartE2EDuration="1.431017741s" totalImagesPullingTime="306.832529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:29.427310459 +0000 UTC m=+1328.637250091" watchObservedRunningTime="2026-10-30 21:31:29.431017741 +0000 UTC m=+1328.640957373" Oct 30 21:31:30.412261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:30.412222 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:31.617475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:31.617433 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:32.198184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:32.197945 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:32.654642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:32.654602 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:33.786592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:33.786557 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:35.025610 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.025580 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw"] Oct 30 21:31:35.030074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.030058 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.047240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.047218 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:31:35.047350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.047280 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:31:35.047350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.047285 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:31:35.054901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.054876 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l5v2k\"" Oct 30 21:31:35.058356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.058331 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw"] Oct 30 21:31:35.103383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103349 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.103559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103492 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwcn\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.103602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103559 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.103642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103621 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.103717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103695 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.103776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103761 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.103892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.103881 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.205704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.205668 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwcn\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.205842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.205759 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.205842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.205802 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.205842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.205832 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.205999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.205896 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.205999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.205986 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.206102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.206057 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.206566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.206536 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.208260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.208236 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.208557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.208530 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.209738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.209714 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.209809 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.209733 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.222575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.222553 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.222853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.222833 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwcn\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mqngw\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.342361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.342282 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:35.400030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.399998 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:35.449946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.445437 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:31:35.659882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:35.659852 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw"] Oct 30 21:31:35.660696 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:31:35.660668 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8beafa_7a51_4882_86cc_6e5e97e57953.slice/crio-8451e9841aef9ce4b1d02e7d64e48eaf5772f5e8a37c81b8e9b365d799af850f WatchSource:0}: Error finding container 8451e9841aef9ce4b1d02e7d64e48eaf5772f5e8a37c81b8e9b365d799af850f: Status 404 returned error can't find the container with id 8451e9841aef9ce4b1d02e7d64e48eaf5772f5e8a37c81b8e9b365d799af850f Oct 30 21:31:36.457157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:36.457122 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" event={"ID":"9e8beafa-7a51-4882-86cc-6e5e97e57953","Type":"ContainerStarted","Data":"8451e9841aef9ce4b1d02e7d64e48eaf5772f5e8a37c81b8e9b365d799af850f"} Oct 30 21:31:37.102310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:37.102267 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:38.199468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.199431 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:31:38.199725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.199516 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:31:38.472664 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.472568 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" event={"ID":"9e8beafa-7a51-4882-86cc-6e5e97e57953","Type":"ContainerStarted","Data":"234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41"} Oct 30 21:31:38.472809 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.472730 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:38.528945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.528897 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" podStartSLOduration=1.993481823 podStartE2EDuration="4.52888377s" totalImagesPullingTime="2.535401947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:38.518648193 +0000 UTC m=+1337.728587823" watchObservedRunningTime="2026-10-30 21:31:38.52888377 +0000 UTC m=+1337.738823399" Oct 30 21:31:38.679284 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.679244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:38.896469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:38.896384 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:39.094987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:39.094954 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:39.279369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:39.279267 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:39.482440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:39.482410 5039 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mqngw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:31:39.482599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:39.482455 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" podUID="9e8beafa-7a51-4882-86cc-6e5e97e57953" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:31:39.902956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:39.902886 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:31:40.060995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:40.060959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:40.770114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:40.770080 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:40.937411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:40.937281 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.089961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:41.089869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:41.675882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:41.675851 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j"] Oct 30 21:31:41.881846 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:41.881807 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:42.025297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:42.025205 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:42.481799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:42.481760 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:31:43.483514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:43.483471 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:44.510630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:44.510597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:44.662502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:44.662470 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:45.646776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:45.646741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:46.676900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:46.676854 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" podUID="81d1a9bd-497f-49c2-879e-a085491c8ea1" containerName="payload-processing" containerID="cri-o://0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f" gracePeriod=25 Oct 30 21:31:46.956624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:46.956599 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:46.999145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:46.999114 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5\") pod \"81d1a9bd-497f-49c2-879e-a085491c8ea1\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " Oct 30 21:31:46.999302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:46.999204 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume\") pod \"81d1a9bd-497f-49c2-879e-a085491c8ea1\" (UID: \"81d1a9bd-497f-49c2-879e-a085491c8ea1\") " Oct 30 21:31:46.999649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:46.999619 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume" pod "81d1a9bd-497f-49c2-879e-a085491c8ea1" (UID: "81d1a9bd-497f-49c2-879e-a085491c8ea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:47.000098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.000066 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d1a9bd-497f-49c2-879e-a085491c8ea1-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:47.001328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.001292 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5" pod "81d1a9bd-497f-49c2-879e-a085491c8ea1" (UID: "81d1a9bd-497f-49c2-879e-a085491c8ea1"). InnerVolumeSpecName "kube-api-access-wtwm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:47.101358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.101328 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/81d1a9bd-497f-49c2-879e-a085491c8ea1-kube-api-access-wtwm5\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:47.147448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.147380 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:47.317631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.317540 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:47.546335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.546303 5039 generic.go:356] "Generic (PLEG): container finished" podID="81d1a9bd-497f-49c2-879e-a085491c8ea1" containerID="0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f" exitCode=0 Oct 30 21:31:47.546549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.546361 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" Oct 30 21:31:47.546549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.546385 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" event={"ID":"81d1a9bd-497f-49c2-879e-a085491c8ea1","Type":"ContainerDied","Data":"0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f"} Oct 30 21:31:47.546549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.546452 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j" event={"ID":"81d1a9bd-497f-49c2-879e-a085491c8ea1","Type":"ContainerDied","Data":"ecf84ca27ecf51351d2bb025a7fd0f48d1bc56d796f07785190fc0c12e8553b7"} Oct 30 21:31:47.546549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.546474 5039 scope.go:122] "RemoveContainer" containerID="0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f" Oct 30 21:31:47.568114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.568091 5039 scope.go:122] "RemoveContainer" containerID="0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f" Oct 30 21:31:47.568548 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:47.568522 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f\": container with ID starting with 0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f not found: ID does not exist" containerID="0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f" Oct 30 21:31:47.568679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.568557 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f"} err="failed to get container status \"0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f\": rpc error: code = NotFound desc = could not find container \"0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f\": container with ID starting with 0d9349d8787f984f5c19e524ffa6a681eae6c4425dd484de3435ff1875a8784f not found: ID does not exist" Oct 30 21:31:47.594909 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.594881 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j"] Oct 30 21:31:47.603197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.603172 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-5fb74b8d77-k9r8j"] Oct 30 21:31:47.947927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:47.947888 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:48.922314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:48.922280 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d1a9bd-497f-49c2-879e-a085491c8ea1" path="/var/lib/kubelet/pods/81d1a9bd-497f-49c2-879e-a085491c8ea1/volumes" Oct 30 21:31:49.701434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:49.700891 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:50.904670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:50.904637 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:51.096998 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:51.096961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:51.510742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:51.510709 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:53.064127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.064093 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp"] Oct 30 21:31:53.064646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.064333 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" podUID="7063944a-7a60-48d0-953f-ebcfbdf20f92" containerName="maas-api" containerID="cri-o://409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62" gracePeriod=30 Oct 30 21:31:53.171716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.171684 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:53.450073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.450038 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:31:53.588679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.588645 5039 generic.go:356] "Generic (PLEG): container finished" podID="7063944a-7a60-48d0-953f-ebcfbdf20f92" containerID="409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62" exitCode=0 Oct 30 21:31:53.588873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.588708 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" Oct 30 21:31:53.588873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.588732 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" event={"ID":"7063944a-7a60-48d0-953f-ebcfbdf20f92","Type":"ContainerDied","Data":"409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62"} Oct 30 21:31:53.588873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.588798 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp" event={"ID":"7063944a-7a60-48d0-953f-ebcfbdf20f92","Type":"ContainerDied","Data":"656c69b0056da2a82e79a8550c151549d6b2978734aa9cf6278ef74a0c4f5783"} Oct 30 21:31:53.588873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.588823 5039 scope.go:122] "RemoveContainer" containerID="409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62" Oct 30 21:31:53.599162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.599146 5039 scope.go:122] "RemoveContainer" containerID="409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62" Oct 30 21:31:53.599474 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:53.599449 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62\": container with ID starting with 409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62 not found: ID does not exist" containerID="409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62" Oct 30 21:31:53.599547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.599486 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62"} err="failed to get container status \"409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62\": rpc error: code = NotFound desc = could not find container \"409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62\": container with ID starting with 409306eacef9fd04bb05325a3b75e57578ac2759e5a491bd00fe88e2630d9d62 not found: ID does not exist" Oct 30 21:31:53.599783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.599767 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4\" (UniqueName: \"kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4\") pod \"7063944a-7a60-48d0-953f-ebcfbdf20f92\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " Oct 30 21:31:53.599833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.599810 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\") pod \"7063944a-7a60-48d0-953f-ebcfbdf20f92\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " Oct 30 21:31:53.599900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.599889 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs\") pod \"7063944a-7a60-48d0-953f-ebcfbdf20f92\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " Oct 30 21:31:53.600059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.600023 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca\") pod \"7063944a-7a60-48d0-953f-ebcfbdf20f92\" (UID: \"7063944a-7a60-48d0-953f-ebcfbdf20f92\") " Oct 30 21:31:53.600865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.600809 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca" pod "7063944a-7a60-48d0-953f-ebcfbdf20f92" (UID: "7063944a-7a60-48d0-953f-ebcfbdf20f92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:53.604419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.602205 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4" pod "7063944a-7a60-48d0-953f-ebcfbdf20f92" (UID: "7063944a-7a60-48d0-953f-ebcfbdf20f92"). InnerVolumeSpecName "kube-api-access-zlpm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:53.604419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.602586 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls" pod "7063944a-7a60-48d0-953f-ebcfbdf20f92" (UID: "7063944a-7a60-48d0-953f-ebcfbdf20f92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:53.604419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.602765 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs" pod "7063944a-7a60-48d0-953f-ebcfbdf20f92" (UID: "7063944a-7a60-48d0-953f-ebcfbdf20f92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:53.702204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.702108 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:53.702204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.702153 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7063944a-7a60-48d0-953f-ebcfbdf20f92-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:53.702204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.702168 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlpm4\" (UniqueName: \"kubernetes.io/projected/7063944a-7a60-48d0-953f-ebcfbdf20f92-kube-api-access-zlpm4\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:53.702204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.702181 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7063944a-7a60-48d0-953f-ebcfbdf20f92-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:31:53.831075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.831039 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:53.932492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.932465 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp"] Oct 30 21:31:53.942353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:53.942329 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-342fc9-6467758b99-rrcwp"] Oct 30 21:31:54.040515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:54.040421 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.221867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:54.221832 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.453002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:54.452965 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.623121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:54.623078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.923785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:54.923749 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7063944a-7a60-48d0-953f-ebcfbdf20f92" path="/var/lib/kubelet/pods/7063944a-7a60-48d0-953f-ebcfbdf20f92/volumes" Oct 30 21:31:55.186631 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:31:55.186547 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:31:55.908694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:55.907978 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:57.209438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:57.209385 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:57.413358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:57.413320 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:58.471286 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:58.471246 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:58.673702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:58.673667 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:59.044539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:31:59.044508 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:00.239884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:00.239846 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:00.393433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:00.393401 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.266927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:01.266892 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.429516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:01.429476 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.580521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:01.580428 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.733529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:01.733373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.761149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:01.761115 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:32:02.617666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:02.617622 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:03.146445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:03.146405 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:03.352432 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:03.352378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:04.131230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:04.131182 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:05.417635 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:05.417604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:05.576446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:05.576405 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.908637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:06.908593 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:07.074732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:07.074683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:08.718265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:08.718227 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:08.876477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:08.876442 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:10.502009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:10.501973 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:12.112188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:12.112148 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:12.856882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:12.856849 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.686594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:13.686552 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.873864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:13.873826 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:14.357789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:14.357752 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:14.502973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:14.502930 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:15.589359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:15.589327 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:17.653745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:17.653706 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:17.799250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:17.799215 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:18.237312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:18.237273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:18.436180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:18.436127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:19.811290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:19.811255 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.660646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:20.660609 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.811946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:20.811882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:21.570504 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:21.570383 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:22.949759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:22.949724 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:23.503150 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.503115 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx"] Oct 30 21:32:23.503810 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.503786 5039 cpu_manager.go:414] "containerMap: removing container" podUID="7063944a-7a60-48d0-953f-ebcfbdf20f92" containerName="maas-api" Oct 30 21:32:23.503810 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.503809 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063944a-7a60-48d0-953f-ebcfbdf20f92" containerName="maas-api" Oct 30 21:32:23.503979 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.503818 5039 cpu_manager.go:414] "containerMap: removing container" podUID="81d1a9bd-497f-49c2-879e-a085491c8ea1" containerName="payload-processing" Oct 30 21:32:23.503979 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.503826 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d1a9bd-497f-49c2-879e-a085491c8ea1" containerName="payload-processing" Oct 30 21:32:23.504078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.503987 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063944a-7a60-48d0-953f-ebcfbdf20f92" containerName="maas-api" Oct 30 21:32:23.504078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.504004 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d1a9bd-497f-49c2-879e-a085491c8ea1" containerName="payload-processing" Oct 30 21:32:23.509101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.509078 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.526890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.526210 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e38f60-openshift-default-dockercfg-nmnq5\"" Oct 30 21:32:23.544319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.543108 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx"] Oct 30 21:32:23.614799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.614765 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.614948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.614847 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.614948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.614932 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.615097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615005 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:23.615097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615051 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.615097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615089 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfpk\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.615254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615200 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.615369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615350 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.615438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615385 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.615513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.615493 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.716773 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716742 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.716930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716778 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.716930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716795 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.716930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716824 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.716930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716916 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716944 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfpk\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.716989 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717091 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717129 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717273 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717316 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717345 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.717835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717597 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.718254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.717623 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.719763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.719737 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.719935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.719914 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.737508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.737483 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.739974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.739949 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfpk\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk\") pod \"e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.759212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.759160 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:32:23.827416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.826870 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:23.926411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:23.923313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.084351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.084255 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.230234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.230205 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx"] Oct 30 21:32:24.231267 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:32:24.231238 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa063257_509c_4fb0_b726_9c9d5912f6f9.slice/crio-17f92a081b4ebc41b8aa24be3866cd6519784c12681836ca9de773e40c56f7ae WatchSource:0}: Error finding container 17f92a081b4ebc41b8aa24be3866cd6519784c12681836ca9de773e40c56f7ae: Status 404 returned error can't find the container with id 17f92a081b4ebc41b8aa24be3866cd6519784c12681836ca9de773e40c56f7ae Oct 30 21:32:24.233663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.233627 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:32:24.233762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.233693 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:32:24.233762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.233722 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:32:24.831110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.831072 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" event={"ID":"fa063257-509c-4fb0-b726-9c9d5912f6f9","Type":"ContainerStarted","Data":"928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941"} Oct 30 21:32:24.831278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.831134 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" event={"ID":"fa063257-509c-4fb0-b726-9c9d5912f6f9","Type":"ContainerStarted","Data":"17f92a081b4ebc41b8aa24be3866cd6519784c12681836ca9de773e40c56f7ae"} Oct 30 21:32:24.896126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:24.896079 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podStartSLOduration=1.896066314 podStartE2EDuration="1.896066314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:24.884360023 +0000 UTC m=+1384.094299653" watchObservedRunningTime="2026-10-30 21:32:24.896066314 +0000 UTC m=+1384.106005945" Oct 30 21:32:25.117964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:25.117863 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:25.190975 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:32:25.190935 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:32:25.826637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:25.826595 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:25.827442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:25.827426 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:25.832542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:25.832520 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:25.840656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:25.840629 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:25.841478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:25.841453 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:32:26.264656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.264619 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b"] Oct 30 21:32:26.273422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.270915 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.290363 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.285524 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0a1ee3-serving-cert\"" Oct 30 21:32:26.298410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.295896 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b"] Oct 30 21:32:26.362408 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.362353 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.362677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.362652 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.362969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.362948 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.363120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.363101 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcfv\" (UniqueName: \"kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.467412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.467042 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.467412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.467085 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.467412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.467189 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.467412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.467232 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcfv\" (UniqueName: \"kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.467774 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:32:26.467712 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-serving-cert: secret "maas-api-e2e-ipp-a-0a1ee3-serving-cert" not found Oct 30 21:32:26.467842 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:32:26.467776 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls podName:0d22a262-62f4-4a57-adab-9f7e5269c967 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:26.967761326 +0000 UTC m=+1386.177700940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls") pod "maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" (UID: "0d22a262-62f4-4a57-adab-9f7e5269c967") : secret "maas-api-e2e-ipp-a-0a1ee3-serving-cert" not found Oct 30 21:32:26.471407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.468088 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.471407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.470093 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.498410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.498129 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxcfv\" (UniqueName: \"kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.535083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.534948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:26.826132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.826043 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:26.973791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.973759 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:26.976372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:26.976343 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:27.092014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.091924 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:27.226692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.226664 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:27.545047 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:32:27.545017 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d22a262_62f4_4a57_adab_9f7e5269c967.slice/crio-c68eae3327f91ee00d7f71e4a7f573964ada055161467110aa78c0dd372b9625 WatchSource:0}: Error finding container c68eae3327f91ee00d7f71e4a7f573964ada055161467110aa78c0dd372b9625: Status 404 returned error can't find the container with id c68eae3327f91ee00d7f71e4a7f573964ada055161467110aa78c0dd372b9625 Oct 30 21:32:27.545446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.545257 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b"] Oct 30 21:32:27.547070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.547049 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:32:27.735873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.735836 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:27.874470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.872176 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" event={"ID":"0d22a262-62f4-4a57-adab-9f7e5269c967","Type":"ContainerStarted","Data":"c68eae3327f91ee00d7f71e4a7f573964ada055161467110aa78c0dd372b9625"} Oct 30 21:32:27.938054 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:27.938013 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:28.879888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:28.879852 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" event={"ID":"0d22a262-62f4-4a57-adab-9f7e5269c967","Type":"ContainerStarted","Data":"74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf"} Oct 30 21:32:28.880313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:28.880010 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:28.915521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:28.915468 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" podStartSLOduration=2.50939797 podStartE2EDuration="2.915454306s" totalImagesPullingTime="406.056336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:28.914711968 +0000 UTC m=+1388.124651599" watchObservedRunningTime="2026-10-30 21:32:28.915454306 +0000 UTC m=+1388.125393936" Oct 30 21:32:28.962074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:28.962038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:29.196843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:29.196809 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml"] Oct 30 21:32:29.197070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:29.197032 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" containerID="cri-o://6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c" gracePeriod=30 Oct 30 21:32:30.065613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:30.065580 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:30.218318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:30.218285 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:30.941121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:30.941090 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:31.130422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:31.130358 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:32.693207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:32.693117 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:32.844545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:32.844507 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:32.992228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:32.992101 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:34.466125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.466076 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:32:34.573816 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.573781 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.573965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.573855 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.573965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.573932 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.573993 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574053 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574084 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574116 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:34.574182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574134 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574161 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:34.574381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574257 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574299 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert\") pod \"1e826dbd-077f-4a18-93ef-b5398198f753\" (UID: \"1e826dbd-077f-4a18-93ef-b5398198f753\") " Oct 30 21:32:34.574529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574377 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:34.574645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574620 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:34.575011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.574974 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.575011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.575007 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.575174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.575021 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.575174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.575035 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.575174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.575035 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:34.576804 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.576770 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "kube-api-access-77w88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:34.576903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.576803 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:34.576903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.576891 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:34.578260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.578240 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token" pod "1e826dbd-077f-4a18-93ef-b5398198f753" (UID: "1e826dbd-077f-4a18-93ef-b5398198f753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:34.589262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.589234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:34.676562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.676537 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e826dbd-077f-4a18-93ef-b5398198f753-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.676562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.676561 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e826dbd-077f-4a18-93ef-b5398198f753-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.676695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.676572 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77w88\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-kube-api-access-77w88\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.676695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.676582 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e826dbd-077f-4a18-93ef-b5398198f753-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.676695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.676590 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e826dbd-077f-4a18-93ef-b5398198f753-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:32:34.891459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.891433 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:32:34.927484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.927438 5039 generic.go:356] "Generic (PLEG): container finished" podID="1e826dbd-077f-4a18-93ef-b5398198f753" containerID="6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c" exitCode=0 Oct 30 21:32:34.927677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.927560 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" Oct 30 21:32:34.928856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.928826 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" event={"ID":"1e826dbd-077f-4a18-93ef-b5398198f753","Type":"ContainerDied","Data":"6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c"} Oct 30 21:32:34.929020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.928879 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml" event={"ID":"1e826dbd-077f-4a18-93ef-b5398198f753","Type":"ContainerDied","Data":"f348fc2cd7160179ff66936c28cbe6bc7fbc13bce18869468e0e2979322bdd49"} Oct 30 21:32:34.929020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.928908 5039 scope.go:122] "RemoveContainer" containerID="6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c" Oct 30 21:32:34.951385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.951360 5039 scope.go:122] "RemoveContainer" containerID="6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c" Oct 30 21:32:34.951829 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:32:34.951806 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c\": container with ID starting with 6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c not found: ID does not exist" containerID="6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c" Oct 30 21:32:34.951907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.951839 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c"} err="failed to get container status \"6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c\": rpc error: code = NotFound desc = could not find container \"6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c\": container with ID starting with 6ba8e8d74a3959e126d5511472fc94fb59870bdac71d8db82a0a0f1086a83b2c not found: ID does not exist" Oct 30 21:32:34.982991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.980463 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml"] Oct 30 21:32:34.986571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:34.986541 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-rcxml"] Oct 30 21:32:36.207737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:36.207692 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:36.921775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:36.921741 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" path="/var/lib/kubelet/pods/1e826dbd-077f-4a18-93ef-b5398198f753/volumes" Oct 30 21:32:37.741003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:37.740974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:39.636073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.636041 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l"] Oct 30 21:32:39.636678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.636661 5039 cpu_manager.go:414] "containerMap: removing container" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" Oct 30 21:32:39.636724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.636679 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" Oct 30 21:32:39.636772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.636763 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e826dbd-077f-4a18-93ef-b5398198f753" containerName="istio-proxy" Oct 30 21:32:39.641789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.641771 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.658140 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.658116 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l"] Oct 30 21:32:39.665074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.665050 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7222e2-serving-cert\"" Oct 30 21:32:39.759719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.759683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:39.836004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.835968 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.836190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.836118 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.836190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.836169 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5cr7\" (UniqueName: \"kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.836277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.836251 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.891145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.891065 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:39.937712 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.937678 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.937900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.937724 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5cr7\" (UniqueName: \"kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.937900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.937772 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.938026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.937912 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.938488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.938460 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.940268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.940235 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.940369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.940305 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.954007 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.953979 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5cr7\" (UniqueName: \"kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7\") pod \"maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:39.954655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:39.954631 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:40.033165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:40.032856 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:40.251685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:40.251654 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l"] Oct 30 21:32:40.255841 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:32:40.255814 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb35621f_6627_4667_9b2b_b5e5adc7d654.slice/crio-68dc97f7a16b41592df5bcdda2c2250977ea699afbb35b4c4bea359fb65aadb6 WatchSource:0}: Error finding container 68dc97f7a16b41592df5bcdda2c2250977ea699afbb35b4c4bea359fb65aadb6: Status 404 returned error can't find the container with id 68dc97f7a16b41592df5bcdda2c2250977ea699afbb35b4c4bea359fb65aadb6 Oct 30 21:32:40.969060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:40.968972 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" event={"ID":"bb35621f-6627-4667-9b2b-b5e5adc7d654","Type":"ContainerStarted","Data":"3d9e39cfd345c611e0827799383a1f871ead6c89705e8a32dd32b7e1f57dd1fb"} Oct 30 21:32:40.969060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:40.969033 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" event={"ID":"bb35621f-6627-4667-9b2b-b5e5adc7d654","Type":"ContainerStarted","Data":"68dc97f7a16b41592df5bcdda2c2250977ea699afbb35b4c4bea359fb65aadb6"} Oct 30 21:32:40.969532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:40.969093 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:41.003053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:41.003007 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" podStartSLOduration=1.597852181 podStartE2EDuration="2.002993843s" totalImagesPullingTime="405.141662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:41.002141742 +0000 UTC m=+1400.212081371" watchObservedRunningTime="2026-10-30 21:32:41.002993843 +0000 UTC m=+1400.212933492" Oct 30 21:32:41.411384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:41.411353 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:43.023796 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:43.023755 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:44.366199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:44.366165 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:44.500734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:44.500685 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:44.634161 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:44.634075 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:44.790165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:44.790130 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:46.729709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:46.729674 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:46.981012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:46.980932 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:32:49.279646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:49.279610 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:49.415854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:49.415821 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:50.575128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:50.575086 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:51.383974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:51.383926 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:53.250278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:53.250244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:54.959008 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:54.958975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:55.091922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:55.091886 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:55.186037 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:32:55.185999 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:32:56.798331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:56.798291 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:58.364317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:58.364281 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.031296 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:59.031262 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.166532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:59.166496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:59.302021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:32:59.301874 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:00.296062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:00.296023 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:01.049001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:01.048942 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:33:02.274799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:02.274762 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:04.177415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:04.172927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:06.406838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:06.406802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:08.378912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:08.378874 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.396160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:09.396104 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.559322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:09.559264 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.754308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:09.754176 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:09.997190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:09.993962 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:11.137653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:11.137615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:11.362896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:11.361856 5039 patch_prober.go:29] interesting pod/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": read tcp 10.132.0.2:34132->10.132.0.102:15021: read: connection reset by peer" start-of-body= Oct 30 21:33:11.362896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:11.361921 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": read tcp 10.132.0.2:34132->10.132.0.102:15021: read: connection reset by peer" Oct 30 21:33:12.198344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.198308 5039 generic.go:356] "Generic (PLEG): container finished" podID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerID="928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941" exitCode=0 Oct 30 21:33:12.198854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.198366 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" event={"ID":"fa063257-509c-4fb0-b726-9c9d5912f6f9","Type":"ContainerDied","Data":"928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941"} Oct 30 21:33:12.198923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.198852 5039 scope.go:122] "RemoveContainer" containerID="928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941" Oct 30 21:33:12.200531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.200501 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:33:12.200605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.200578 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:33:12.200645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.200612 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:33:12.813859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.813816 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:12.983204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:12.983162 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.208320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:13.208281 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" event={"ID":"fa063257-509c-4fb0-b726-9c9d5912f6f9","Type":"ContainerStarted","Data":"7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02"} Oct 30 21:33:13.827473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:13.827441 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:33:13.827641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:13.827605 5039 patch_prober.go:29] interesting pod/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:33:13.827695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:13.827655 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:33:14.507502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:14.507467 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:14.827987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:14.827907 5039 patch_prober.go:29] interesting pod/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:33:14.827987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:14.827957 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:33:15.827449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:15.827418 5039 patch_prober.go:29] interesting pod/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:33:15.827808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:15.827471 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:33:16.590905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:16.590863 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:16.827983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:16.827947 5039 patch_prober.go:29] interesting pod/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:33:16.828358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:16.828006 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:33:17.827960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:17.827925 5039 patch_prober.go:29] interesting pod/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:33:17.828134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:17.827978 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:33:18.423046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:18.422997 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:18.831506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:18.831429 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:33:18.831855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:18.831814 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:33:18.832496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:18.832476 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:33:19.720604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:19.720573 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:19.864358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:19.864322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:21.020481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:21.020373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:22.627410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:22.627328 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:22.975496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:22.975457 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:33:23.709626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:23.709586 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:23.847649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:23.847607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:23.991828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:23.991733 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:25.023521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:25.023443 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:25.184578 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:33:25.184541 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:33:26.806815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:26.806784 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:26.947919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:26.947877 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:27.525067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:27.525026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:27.785333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:27.785244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:28.720564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:28.720528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:30.577577 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:30.577530 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:31.438157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:31.438121 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:31.599351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:31.599315 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:32.327662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:32.327621 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:33.127576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:33.127531 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:33.276439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:33.276378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:34.546040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:34.546008 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:35.700422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:35.700348 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:35.856708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:35.856643 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:36.295268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:36.295227 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:36.857954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:36.857918 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:37.019238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:37.019204 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:37.159336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:37.159299 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:38.840010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:38.839975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:38.989793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:38.989757 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:39.397871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:39.397838 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:40.952138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:40.952107 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:41.097595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:41.097566 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:41.597251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:41.597222 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:42.713604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:42.713561 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:42.860622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:42.860592 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:44.013827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:44.013791 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:45.265549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:45.265512 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:45.409510 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:45.409472 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:46.548792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:46.548758 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:48.666681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:48.666647 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:50.184676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:50.184636 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:51.727952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:51.727905 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:53.396627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:53.396588 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:55.145708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:55.145673 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:55.195885 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:33:55.195826 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:33:55.807333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:55.807297 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:55.948751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:55.948723 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:56.863993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:56.863960 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:57.011901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:57.011870 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:58.771955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:33:58.771922 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:00.442350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:00.442211 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:02.673616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:02.673526 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:04.809741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:04.809701 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:06.785997 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:06.785952 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:08.259279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:08.259242 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:08.396521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:08.396480 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:09.685901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:09.685856 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:10.362509 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:10.362478 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:10.498595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:10.498566 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:10.634331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:10.634239 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:10.974858 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:10.974830 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:12.785674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:12.785634 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:12.880083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:12.869026 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:34:12.932595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:12.932553 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:13.153346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:13.153315 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:15.255668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:15.255636 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:16.919684 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:16.919644 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:17.068852 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:17.068804 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:17.257994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:17.257909 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:17.466991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:17.466955 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:19.307542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:19.307499 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:20.969569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:20.969532 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:21.122486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:21.122454 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:34:21.127499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:21.127467 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:34:22.578726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:22.578692 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:24.220733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:24.220692 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:25.184812 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:34:25.184777 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:34:25.811095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:25.811066 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.113119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:29.113078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.293536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:29.293494 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.447383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:29.447343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.804681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:29.804520 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:29.985919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:29.985882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:30.257499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:30.257465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:30.590984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:30.590897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:32.555984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:32.555951 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:32.823106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:32.823026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:32.973857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:32.973827 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:33.113567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:33.113478 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:34.773722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:34.773684 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:36.410758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:36.410725 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:36.561952 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:36.561906 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:36.843370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:36.843299 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:37.743416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:37.743353 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt"] Oct 30 21:34:37.744203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:37.744166 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" podUID="06dcdf53-89b6-4302-94ef-d59d05d1f718" containerName="maas-api" containerID="cri-o://14fa04e59f63586bac833682d79382912da37de752b41f42c108c39c006efba9" gracePeriod=30 Oct 30 21:34:37.850962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:37.850884 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:37.892668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:37.892637 5039 generic.go:356] "Generic (PLEG): container finished" podID="06dcdf53-89b6-4302-94ef-d59d05d1f718" containerID="14fa04e59f63586bac833682d79382912da37de752b41f42c108c39c006efba9" exitCode=0 Oct 30 21:34:37.892825 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:37.892676 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" event={"ID":"06dcdf53-89b6-4302-94ef-d59d05d1f718","Type":"ContainerDied","Data":"14fa04e59f63586bac833682d79382912da37de752b41f42c108c39c006efba9"} Oct 30 21:34:38.032383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.032298 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:38.069919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.069889 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:34:38.113234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.112625 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls\") pod \"06dcdf53-89b6-4302-94ef-d59d05d1f718\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " Oct 30 21:34:38.113234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.112815 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b\" (UniqueName: \"kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b\") pod \"06dcdf53-89b6-4302-94ef-d59d05d1f718\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " Oct 30 21:34:38.113234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.112847 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca\") pod \"06dcdf53-89b6-4302-94ef-d59d05d1f718\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " Oct 30 21:34:38.113234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.112884 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs\") pod \"06dcdf53-89b6-4302-94ef-d59d05d1f718\" (UID: \"06dcdf53-89b6-4302-94ef-d59d05d1f718\") " Oct 30 21:34:38.113862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.113834 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca" pod "06dcdf53-89b6-4302-94ef-d59d05d1f718" (UID: "06dcdf53-89b6-4302-94ef-d59d05d1f718"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:38.116649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.116627 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls" pod "06dcdf53-89b6-4302-94ef-d59d05d1f718" (UID: "06dcdf53-89b6-4302-94ef-d59d05d1f718"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:38.118211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.118193 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs" pod "06dcdf53-89b6-4302-94ef-d59d05d1f718" (UID: "06dcdf53-89b6-4302-94ef-d59d05d1f718"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:38.119513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.119489 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b" pod "06dcdf53-89b6-4302-94ef-d59d05d1f718" (UID: "06dcdf53-89b6-4302-94ef-d59d05d1f718"). InnerVolumeSpecName "kube-api-access-7br8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:38.175949 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.175914 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:38.214731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.214705 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7br8b\" (UniqueName: \"kubernetes.io/projected/06dcdf53-89b6-4302-94ef-d59d05d1f718-kube-api-access-7br8b\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:38.214731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.214731 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dcdf53-89b6-4302-94ef-d59d05d1f718-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:38.214850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.214741 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:38.214850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.214749 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06dcdf53-89b6-4302-94ef-d59d05d1f718-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:38.391166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.391133 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:38.634662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.634630 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:38.819293 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.819211 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:38.901045 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.901016 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" event={"ID":"06dcdf53-89b6-4302-94ef-d59d05d1f718","Type":"ContainerDied","Data":"9beeb6792dc750d41b92ed0086a51884da75a8464313c9bc08923195906c6df2"} Oct 30 21:34:38.901218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.901023 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt" Oct 30 21:34:38.901218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.901080 5039 scope.go:122] "RemoveContainer" containerID="14fa04e59f63586bac833682d79382912da37de752b41f42c108c39c006efba9" Oct 30 21:34:38.945354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.945324 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt"] Oct 30 21:34:38.954802 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:38.954776 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c027ae-9b4b4dc44-hwkbt"] Oct 30 21:34:39.219168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:39.219141 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:40.921232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:40.921202 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dcdf53-89b6-4302-94ef-d59d05d1f718" path="/var/lib/kubelet/pods/06dcdf53-89b6-4302-94ef-d59d05d1f718/volumes" Oct 30 21:34:41.245127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:41.245046 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:42.276056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:42.276024 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:42.413579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:42.413547 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:42.554546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:42.554459 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:43.701448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:43.701403 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:45.009379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:45.009344 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:45.175148 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:45.175112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:45.380647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:45.380569 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:45.590714 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:45.590679 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:47.210345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:47.210309 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:48.982560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:48.982529 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:50.336594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:50.336562 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:34:50.762584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:50.762534 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:51.922871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:51.922836 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:52.085772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:52.085735 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:53.322405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:53.322351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:54.691688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:54.691654 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l"] Oct 30 21:34:54.692220 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:54.691915 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" podUID="bb35621f-6627-4667-9b2b-b5e5adc7d654" containerName="maas-api" containerID="cri-o://3d9e39cfd345c611e0827799383a1f871ead6c89705e8a32dd32b7e1f57dd1fb" gracePeriod=30 Oct 30 21:34:54.827657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:54.827625 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.005754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.005725 5039 generic.go:356] "Generic (PLEG): container finished" podID="bb35621f-6627-4667-9b2b-b5e5adc7d654" containerID="3d9e39cfd345c611e0827799383a1f871ead6c89705e8a32dd32b7e1f57dd1fb" exitCode=0 Oct 30 21:34:55.005886 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.005807 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" event={"ID":"bb35621f-6627-4667-9b2b-b5e5adc7d654","Type":"ContainerDied","Data":"3d9e39cfd345c611e0827799383a1f871ead6c89705e8a32dd32b7e1f57dd1fb"} Oct 30 21:34:55.005886 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.005869 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" event={"ID":"bb35621f-6627-4667-9b2b-b5e5adc7d654","Type":"ContainerDied","Data":"68dc97f7a16b41592df5bcdda2c2250977ea699afbb35b4c4bea359fb65aadb6"} Oct 30 21:34:55.005958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.005892 5039 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68dc97f7a16b41592df5bcdda2c2250977ea699afbb35b4c4bea359fb65aadb6" Oct 30 21:34:55.019051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.019032 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:34:55.024772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.024743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.058012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.057983 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs\") pod \"bb35621f-6627-4667-9b2b-b5e5adc7d654\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " Oct 30 21:34:55.058148 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.058054 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7\" (UniqueName: \"kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7\") pod \"bb35621f-6627-4667-9b2b-b5e5adc7d654\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " Oct 30 21:34:55.058201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.058150 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls\") pod \"bb35621f-6627-4667-9b2b-b5e5adc7d654\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " Oct 30 21:34:55.060354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.060322 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs" pod "bb35621f-6627-4667-9b2b-b5e5adc7d654" (UID: "bb35621f-6627-4667-9b2b-b5e5adc7d654"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:55.060621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.060600 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7" pod "bb35621f-6627-4667-9b2b-b5e5adc7d654" (UID: "bb35621f-6627-4667-9b2b-b5e5adc7d654"). InnerVolumeSpecName "kube-api-access-k5cr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:55.060960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.060825 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls" pod "bb35621f-6627-4667-9b2b-b5e5adc7d654" (UID: "bb35621f-6627-4667-9b2b-b5e5adc7d654"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:55.160230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.160200 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca\") pod \"bb35621f-6627-4667-9b2b-b5e5adc7d654\" (UID: \"bb35621f-6627-4667-9b2b-b5e5adc7d654\") " Oct 30 21:34:55.160547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.160524 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca" pod "bb35621f-6627-4667-9b2b-b5e5adc7d654" (UID: "bb35621f-6627-4667-9b2b-b5e5adc7d654"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:55.160779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.160762 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:55.160837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.160784 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5cr7\" (UniqueName: \"kubernetes.io/projected/bb35621f-6627-4667-9b2b-b5e5adc7d654-kube-api-access-k5cr7\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:55.160837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.160798 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb35621f-6627-4667-9b2b-b5e5adc7d654-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:55.160837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.160811 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb35621f-6627-4667-9b2b-b5e5adc7d654-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:34:55.171051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.171013 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.185940 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:34:55.185909 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:34:55.485595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.485563 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.666546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.666492 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.871647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:55.871552 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:56.011665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:56.011637 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l" Oct 30 21:34:56.057447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:56.057410 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l"] Oct 30 21:34:56.067120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:56.067096 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7222e2-68fcb45b67-mhk9l"] Oct 30 21:34:56.921895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:56.921868 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb35621f-6627-4667-9b2b-b5e5adc7d654" path="/var/lib/kubelet/pods/bb35621f-6627-4667-9b2b-b5e5adc7d654/volumes" Oct 30 21:34:57.851468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:57.851433 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:58.039149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:58.039115 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:59.621169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:34:59.621136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:00.902795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:00.902748 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:01.076760 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:01.076726 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:01.778038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:01.777991 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:01.954652 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:01.954615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:02.111832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:02.111560 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:02.560230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:02.560188 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:02.715661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:02.715620 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:02.867076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:02.866994 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:03.008191 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:03.008151 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:03.234576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:03.234542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:03.389046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:03.389001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:04.920742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:04.920710 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:05.286266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:05.286177 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:05.481204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:05.481166 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:06.856725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:06.856686 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:07.598197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:07.598158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:07.804277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:07.804243 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:08.258353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:08.258314 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:09.745849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:09.745818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:09.939305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:09.939265 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:11.508631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:11.508591 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:11.742993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:11.742920 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:11.952085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:11.952057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:12.149020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:12.148983 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:12.149512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:12.149494 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx"] Oct 30 21:35:12.149901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:12.149867 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" containerID="cri-o://7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02" gracePeriod=30 Oct 30 21:35:12.313315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:12.313228 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.338614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:13.338578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.845747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:13.845711 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:14.187778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:14.187741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:14.468902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:14.468802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:14.666604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:14.666571 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.360352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:16.360311 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.656420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:16.656354 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.877911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:16.877875 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:17.419984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.419934 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:35:17.507611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507585 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.507747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507725 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.507827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507787 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.507827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507819 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.507932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507851 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.507932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507852 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:17.507932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507877 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.507932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507919 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.508112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.507974 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.508112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508064 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket\") pod \"fa063257-509c-4fb0-b726-9c9d5912f6f9\" (UID: \"fa063257-509c-4fb0-b726-9c9d5912f6f9\") " Oct 30 21:35:17.508356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508328 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:17.508508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508359 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:17.508573 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508535 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:17.508627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508570 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:17.508924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508794 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa063257-509c-4fb0-b726-9c9d5912f6f9-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.508924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508820 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.508924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508835 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.508924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508851 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.508924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.508877 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.510508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.510481 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:17.510508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.510491 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:35:17.510676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.510630 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "kube-api-access-2mfpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:17.511378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.511357 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token" pod "fa063257-509c-4fb0-b726-9c9d5912f6f9" (UID: "fa063257-509c-4fb0-b726-9c9d5912f6f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:17.610112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.610087 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.610112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.610109 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.610236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.610118 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:17.610236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:17.610127 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfpk\" (UniqueName: \"kubernetes.io/projected/fa063257-509c-4fb0-b726-9c9d5912f6f9-kube-api-access-2mfpk\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:35:18.166751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.166718 5039 generic.go:356] "Generic (PLEG): container finished" podID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerID="7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02" exitCode=0 Oct 30 21:35:18.166868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.166761 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" Oct 30 21:35:18.166868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.166796 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" event={"ID":"fa063257-509c-4fb0-b726-9c9d5912f6f9","Type":"ContainerDied","Data":"7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02"} Oct 30 21:35:18.166868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.166848 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx" event={"ID":"fa063257-509c-4fb0-b726-9c9d5912f6f9","Type":"ContainerDied","Data":"17f92a081b4ebc41b8aa24be3866cd6519784c12681836ca9de773e40c56f7ae"} Oct 30 21:35:18.166980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.166878 5039 scope.go:122] "RemoveContainer" containerID="7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02" Oct 30 21:35:18.177720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.177705 5039 scope.go:122] "RemoveContainer" containerID="928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941" Oct 30 21:35:18.185994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.185977 5039 scope.go:122] "RemoveContainer" containerID="7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02" Oct 30 21:35:18.186222 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:35:18.186207 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02\": container with ID starting with 7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02 not found: ID does not exist" containerID="7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02" Oct 30 21:35:18.186556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.186233 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02"} err="failed to get container status \"7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02\": rpc error: code = NotFound desc = could not find container \"7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02\": container with ID starting with 7cc81ff700d7128c279386c6d81e4feaaf1cc371f8af55f363af97b011ec4b02 not found: ID does not exist" Oct 30 21:35:18.186556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.186249 5039 scope.go:122] "RemoveContainer" containerID="928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941" Oct 30 21:35:18.186556 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:35:18.186537 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941\": container with ID starting with 928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941 not found: ID does not exist" containerID="928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941" Oct 30 21:35:18.186720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.186559 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941"} err="failed to get container status \"928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941\": rpc error: code = NotFound desc = could not find container \"928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941\": container with ID starting with 928d8ee14de2d2c0a8c6575880c207afa035c6bf6799113109ea37db46e58941 not found: ID does not exist" Oct 30 21:35:18.227515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.227487 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx"] Oct 30 21:35:18.236975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.236955 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e38f60-openshift-default-db6bfcfb5-hgmvx"] Oct 30 21:35:18.598616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.598523 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:18.913970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.913927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:18.922041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:18.922012 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" path="/var/lib/kubelet/pods/fa063257-509c-4fb0-b726-9c9d5912f6f9/volumes" Oct 30 21:35:19.214280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:19.214194 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:19.514067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:19.513983 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:20.680415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:20.680370 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:20.918876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:20.918842 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:21.801332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:21.801291 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:22.896084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:22.896047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:23.135533 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:23.135505 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:24.091983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:24.091949 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:24.788724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:24.788689 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:25.099841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:25.099749 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:25.190153 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:35:25.190111 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:35:25.334410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:25.333796 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:26.899337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:26.899273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.993114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:28.993074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:29.624127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:29.624093 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:35:30.589498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:30.589464 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:32.236438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:32.236385 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:32.398930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:32.398889 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:34.034912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:34.034867 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:35.832451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:35.832417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:37.664990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:37.664941 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:39.838233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:39.838199 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:41.693118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:41.693074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:41.882980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:41.882948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.465617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:43.465579 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.714431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:43.714373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.928413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:43.923258 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:44.177186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:44.177152 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:44.406130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:44.406098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:44.602532 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:44.602496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:44.912623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:44.912577 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:46.604904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:46.604850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:48.849257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:48.849212 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:51.348642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:51.348603 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:55.023683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:55.023649 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:55.183296 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:35:55.183263 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:35:55.294678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:55.294594 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:56.100756 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:56.100725 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:57.105632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:57.105598 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:57.374580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:57.374500 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:58.178428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:35:58.178371 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:00.001181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:00.001146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:00.962707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:00.962666 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:02.072949 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:02.072911 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:02.349521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:02.349420 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:02.690110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:02.686033 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:02.936511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:02.936465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:04.373921 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:04.373888 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:06.374239 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:06.374200 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:06.720769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:06.720732 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:06.963488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:06.963452 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:07.457817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:07.457784 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:36:08.614599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:08.614564 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:09.117217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:09.117183 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:09.621622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:09.621586 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:11.963382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:11.963341 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:13.427122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:13.427081 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:14.209050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:14.209013 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:14.894749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:14.894715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:16.104342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:16.104310 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:16.355079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:16.354941 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:16.987907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:16.987857 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:17.469691 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:17.469630 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:19.018882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:19.018849 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:19.428601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:19.428569 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:20.306871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:20.306828 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:20.691982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:20.691946 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:21.016096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:21.016007 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:21.234553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:21.234516 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:22.574058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:22.574022 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:22.940567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:22.940535 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:25.183232 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:36:25.183197 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:36:25.325549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:25.325518 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:25.743505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:25.743476 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:27.860537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:27.860498 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:28.203758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:28.203721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:30.326385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:30.326352 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:32.275251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:32.275207 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:34.295864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:34.295827 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:34.562465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:34.561151 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:34.966074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:34.966041 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:35.251933 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:35.251835 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:35.684255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:35.684214 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:37.176767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:37.176731 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:39.178071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:39.178038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:41.197677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:41.197633 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:43.056639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:43.056601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:43.671114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:43.671080 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:43.865759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:43.865723 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:44.113945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:44.113838 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:45.012275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:45.012239 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:46.913924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:46.913880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:47.292549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:47.292432 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:47.538003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:47.537965 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:47.855768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:47.855734 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:49.430884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:49.430848 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:50.194212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:50.194181 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:50.403834 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:50.403810 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:51.205936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:51.205907 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:51.394481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:51.394443 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:51.875963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:51.875920 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.030637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:53.030602 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:53.180261 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:53.180208 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:54.161905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:54.161868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:54.366162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:54.366122 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:54.370741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:54.370710 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:36:55.231134 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:36:55.231085 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:36:56.128565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:56.128529 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:58.018919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:58.018881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:59.710799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:36:59.709949 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:01.429446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:01.429321 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:03.101444 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:03.101366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:04.859844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:04.859786 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:06.594574 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:06.594539 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.617312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:07.617274 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:08.287743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:08.287660 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:09.959458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:09.959383 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:11.797811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:11.797779 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls"] Oct 30 21:37:11.798273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:11.797988 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" containerID="cri-o://ac44dce38d3a9f464d50993b0bcc7da1e41188034dd57728b7b159d6e582709b" gracePeriod=30 Oct 30 21:37:11.874928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:11.874891 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:13.517685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:13.517638 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:14.336895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:14.336854 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.433732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:16.433700 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.592921 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:16.592887 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.746481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:16.746378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.942969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:16.942941 5039 generic.go:356] "Generic (PLEG): container finished" podID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerID="ac44dce38d3a9f464d50993b0bcc7da1e41188034dd57728b7b159d6e582709b" exitCode=0 Oct 30 21:37:16.943270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:16.943247 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" event={"ID":"0a3e1041-a476-42d7-bb2c-d5d8b813161f","Type":"ContainerDied","Data":"ac44dce38d3a9f464d50993b0bcc7da1e41188034dd57728b7b159d6e582709b"} Oct 30 21:37:17.135360 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.135337 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:37:17.255294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255222 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255285 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255335 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255377 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255413 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255608 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255684 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:17.255813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255746 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255896 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.255961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255888 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:17.255961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.255947 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert\") pod \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\" (UID: \"0a3e1041-a476-42d7-bb2c-d5d8b813161f\") " Oct 30 21:37:17.256246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256225 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:17.256494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256470 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:17.256623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256604 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.256689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256632 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.256689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256650 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.256689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256665 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.256795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.256726 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:17.257916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.257886 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "kube-api-access-lvmtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:17.258041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.257942 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:17.258217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.258189 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:37:17.258551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.258532 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy" pod "0a3e1041-a476-42d7-bb2c-d5d8b813161f" (UID: "0a3e1041-a476-42d7-bb2c-d5d8b813161f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:17.357699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.357669 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.357699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.357698 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.357699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.357706 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.357927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.357715 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmtc\" (UniqueName: \"kubernetes.io/projected/0a3e1041-a476-42d7-bb2c-d5d8b813161f-kube-api-access-lvmtc\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.357927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.357723 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a3e1041-a476-42d7-bb2c-d5d8b813161f-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.950524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.950497 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" Oct 30 21:37:17.950927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.950532 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls" event={"ID":"0a3e1041-a476-42d7-bb2c-d5d8b813161f","Type":"ContainerDied","Data":"e72b74d37e832b3e285fa82426ba568bfc9ea09e6109672920f3100b57aa45ed"} Oct 30 21:37:17.950927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:17.950582 5039 scope.go:122] "RemoveContainer" containerID="ac44dce38d3a9f464d50993b0bcc7da1e41188034dd57728b7b159d6e582709b" Oct 30 21:37:18.006908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:18.006881 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls"] Oct 30 21:37:18.016308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:18.016289 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-fsrls"] Oct 30 21:37:18.503469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:18.503439 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:18.921713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:18.921676 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" path="/var/lib/kubelet/pods/0a3e1041-a476-42d7-bb2c-d5d8b813161f/volumes" Oct 30 21:37:20.605558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:20.605521 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:21.663677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:21.663647 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:21.827197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:21.827159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:22.739292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:22.739252 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:24.378226 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:24.378190 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:24.959551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:24.959496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:25.173608 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:25.173577 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:25.184633 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:37:25.184602 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:37:25.310424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:25.310307 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:25.689519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:25.689491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:26.393090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:26.393048 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:27.291238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:27.291204 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:29.319505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:29.319470 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:30.391287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:30.391243 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:37:31.040376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:31.040335 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:32.433434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:32.433382 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:32.580630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:32.580594 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:32.779309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:32.779221 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:32.932416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:32.927752 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:34.479258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:34.479220 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:34.631438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:34.631401 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:35.045065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.045034 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw"] Oct 30 21:37:35.045276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.045234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:35.045276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.045257 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" podUID="9e8beafa-7a51-4882-86cc-6e5e97e57953" containerName="discovery" containerID="cri-o://234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41" gracePeriod=30 Oct 30 21:37:35.314301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.314249 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:37:35.383795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.383766 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.383910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.383830 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.383998 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.383983 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.384037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.384023 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.384083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.384058 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.384122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.384100 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.384247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.384224 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap\") pod \"9e8beafa-7a51-4882-86cc-6e5e97e57953\" (UID: \"9e8beafa-7a51-4882-86cc-6e5e97e57953\") " Oct 30 21:37:35.384807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.384752 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:35.384912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.384898 5039 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-ca-configmap\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.387480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.387287 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:35.387480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.387323 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:35.387480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.387348 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:35.387480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.387445 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:35.387743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.387649 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "kube-api-access-2dwcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:35.387917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.387893 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token" pod "9e8beafa-7a51-4882-86cc-6e5e97e57953" (UID: "9e8beafa-7a51-4882-86cc-6e5e97e57953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:35.486319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.486282 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwcn\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-kube-api-access-2dwcn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.486319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.486322 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.486692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.486336 5039 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8beafa-7a51-4882-86cc-6e5e97e57953-local-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.486692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.486350 5039 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-kubeconfig\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.486692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.486363 5039 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-istio-csr-dns-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.486692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:35.486378 5039 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e8beafa-7a51-4882-86cc-6e5e97e57953-cacerts\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:36.064268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.064233 5039 generic.go:356] "Generic (PLEG): container finished" podID="9e8beafa-7a51-4882-86cc-6e5e97e57953" containerID="234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41" exitCode=0 Oct 30 21:37:36.064268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.064269 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" Oct 30 21:37:36.064492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.064317 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" event={"ID":"9e8beafa-7a51-4882-86cc-6e5e97e57953","Type":"ContainerDied","Data":"234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41"} Oct 30 21:37:36.064492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.064354 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw" event={"ID":"9e8beafa-7a51-4882-86cc-6e5e97e57953","Type":"ContainerDied","Data":"8451e9841aef9ce4b1d02e7d64e48eaf5772f5e8a37c81b8e9b365d799af850f"} Oct 30 21:37:36.064492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.064373 5039 scope.go:122] "RemoveContainer" containerID="234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41" Oct 30 21:37:36.075005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.074989 5039 scope.go:122] "RemoveContainer" containerID="234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41" Oct 30 21:37:36.075261 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:37:36.075237 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41\": container with ID starting with 234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41 not found: ID does not exist" containerID="234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41" Oct 30 21:37:36.075340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.075269 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41"} err="failed to get container status \"234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41\": rpc error: code = NotFound desc = could not find container \"234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41\": container with ID starting with 234bf0a8aeb0e26d53709cbaf0041db6ca03c540a6dfc8310d099bb1e0fbec41 not found: ID does not exist" Oct 30 21:37:36.118159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.118137 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw"] Oct 30 21:37:36.127826 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.127800 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mqngw"] Oct 30 21:37:36.899630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.899596 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:36.922185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:36.922148 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8beafa-7a51-4882-86cc-6e5e97e57953" path="/var/lib/kubelet/pods/9e8beafa-7a51-4882-86cc-6e5e97e57953/volumes" Oct 30 21:37:38.615639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:38.615588 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:39.011309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:39.011270 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:39.166370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:39.166338 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:40.775461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:40.774262 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:42.057074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.057035 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b"] Oct 30 21:37:42.057567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.057322 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" podUID="0d22a262-62f4-4a57-adab-9f7e5269c967" containerName="maas-api" containerID="cri-o://74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf" gracePeriod=30 Oct 30 21:37:42.169889 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.169845 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:42.345745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.345714 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:42.430127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.429938 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7"] Oct 30 21:37:42.430654 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.430627 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" containerID="cri-o://4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff" gracePeriod=30 Oct 30 21:37:42.494348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.494253 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:42.521669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.521650 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:37:42.566908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.566862 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs\") pod \"0d22a262-62f4-4a57-adab-9f7e5269c967\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " Oct 30 21:37:42.567201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.567180 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\") pod \"0d22a262-62f4-4a57-adab-9f7e5269c967\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " Oct 30 21:37:42.567520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.567453 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca\") pod \"0d22a262-62f4-4a57-adab-9f7e5269c967\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " Oct 30 21:37:42.567674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.567656 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv\" (UniqueName: \"kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv\") pod \"0d22a262-62f4-4a57-adab-9f7e5269c967\" (UID: \"0d22a262-62f4-4a57-adab-9f7e5269c967\") " Oct 30 21:37:42.569368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.569329 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca" pod "0d22a262-62f4-4a57-adab-9f7e5269c967" (UID: "0d22a262-62f4-4a57-adab-9f7e5269c967"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:42.626471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.615515 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs" pod "0d22a262-62f4-4a57-adab-9f7e5269c967" (UID: "0d22a262-62f4-4a57-adab-9f7e5269c967"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:42.626471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.615636 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv" pod "0d22a262-62f4-4a57-adab-9f7e5269c967" (UID: "0d22a262-62f4-4a57-adab-9f7e5269c967"). InnerVolumeSpecName "kube-api-access-fxcfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:42.667472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.644104 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls" pod "0d22a262-62f4-4a57-adab-9f7e5269c967" (UID: "0d22a262-62f4-4a57-adab-9f7e5269c967"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:42.670744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.669894 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:42.670744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.669941 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d22a262-62f4-4a57-adab-9f7e5269c967-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:42.670744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.669958 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxcfv\" (UniqueName: \"kubernetes.io/projected/0d22a262-62f4-4a57-adab-9f7e5269c967-kube-api-access-fxcfv\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:42.670744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.669973 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d22a262-62f4-4a57-adab-9f7e5269c967-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:42.771491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.771459 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:42.937033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:42.937001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:43.116369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.116336 5039 generic.go:356] "Generic (PLEG): container finished" podID="0d22a262-62f4-4a57-adab-9f7e5269c967" containerID="74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf" exitCode=0 Oct 30 21:37:43.116722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.116405 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" Oct 30 21:37:43.116722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.116421 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" event={"ID":"0d22a262-62f4-4a57-adab-9f7e5269c967","Type":"ContainerDied","Data":"74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf"} Oct 30 21:37:43.116722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.116475 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b" event={"ID":"0d22a262-62f4-4a57-adab-9f7e5269c967","Type":"ContainerDied","Data":"c68eae3327f91ee00d7f71e4a7f573964ada055161467110aa78c0dd372b9625"} Oct 30 21:37:43.116722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.116502 5039 scope.go:122] "RemoveContainer" containerID="74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf" Oct 30 21:37:43.129418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.129383 5039 scope.go:122] "RemoveContainer" containerID="74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf" Oct 30 21:37:43.129702 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:37:43.129677 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf\": container with ID starting with 74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf not found: ID does not exist" containerID="74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf" Oct 30 21:37:43.129767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.129715 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf"} err="failed to get container status \"74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf\": rpc error: code = NotFound desc = could not find container \"74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf\": container with ID starting with 74c7263f8f2f27ed38caa2cf13624f377cc5691638afbaa729d16557764e5ebf not found: ID does not exist" Oct 30 21:37:43.160926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.160902 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b"] Oct 30 21:37:43.170214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.170194 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1ee3-79c7cbb8cf-4km4b"] Oct 30 21:37:43.419478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.419454 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:43.979379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:43.979344 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:44.123167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:44.123124 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:44.735545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:44.735502 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:44.920965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:44.920933 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d22a262-62f4-4a57-adab-9f7e5269c967" path="/var/lib/kubelet/pods/0d22a262-62f4-4a57-adab-9f7e5269c967/volumes" Oct 30 21:37:46.745996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:46.745963 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:47.221522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.221486 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:47.423524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.423493 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:37:47.423678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.423547 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:37:47.709811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.709789 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:37:47.835076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835005 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835048 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835072 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835104 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835168 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835193 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835230 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835330 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835379 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data\") pod \"48af0e61-5d81-4ab2-b651-121bd5fff894\" (UID: \"48af0e61-5d81-4ab2-b651-121bd5fff894\") " Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835479 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:47.835659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835500 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:47.836072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835680 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:47.836072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835847 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:47.836072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835971 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.836072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.835992 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.836072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.836008 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.836072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.836021 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.836379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.836228 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:47.837246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.837211 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:37:47.837367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.837277 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:47.837367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.837319 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:47.837491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.837401 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv" pod "48af0e61-5d81-4ab2-b651-121bd5fff894" (UID: "48af0e61-5d81-4ab2-b651-121bd5fff894"). InnerVolumeSpecName "kube-api-access-wzxdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:47.936817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.936792 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48af0e61-5d81-4ab2-b651-121bd5fff894-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.936817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.936816 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzxdv\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-kube-api-access-wzxdv\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.936974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.936826 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.936974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.936834 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.936974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:47.936842 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48af0e61-5d81-4ab2-b651-121bd5fff894-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:37:48.147913 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.147880 5039 generic.go:356] "Generic (PLEG): container finished" podID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerID="4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff" exitCode=0 Oct 30 21:37:48.148023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.147927 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" Oct 30 21:37:48.148023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.147919 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" event={"ID":"48af0e61-5d81-4ab2-b651-121bd5fff894","Type":"ContainerDied","Data":"4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff"} Oct 30 21:37:48.148118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.148040 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7" event={"ID":"48af0e61-5d81-4ab2-b651-121bd5fff894","Type":"ContainerDied","Data":"6463506efcfe8839c35989259d487c0c3eae45141cad69ca9425018c42fd4d10"} Oct 30 21:37:48.148118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.148067 5039 scope.go:122] "RemoveContainer" containerID="4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff" Oct 30 21:37:48.158431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.158413 5039 scope.go:122] "RemoveContainer" containerID="4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff" Oct 30 21:37:48.158675 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:37:48.158659 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff\": container with ID starting with 4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff not found: ID does not exist" containerID="4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff" Oct 30 21:37:48.158727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.158683 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff"} err="failed to get container status \"4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff\": rpc error: code = NotFound desc = could not find container \"4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff\": container with ID starting with 4959b2cf08f550ed58506c1242bbc228a39ecd7362c826033d51ab5853b76aff not found: ID does not exist" Oct 30 21:37:48.191779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.191757 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7"] Oct 30 21:37:48.201667 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.201645 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-nmcm7"] Oct 30 21:37:48.436203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.436137 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:48.582125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.582091 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:48.919634 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.919590 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:48.922282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.922259 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" path="/var/lib/kubelet/pods/48af0e61-5d81-4ab2-b651-121bd5fff894/volumes" Oct 30 21:37:48.979010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.978980 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx"] Oct 30 21:37:48.979589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979570 5039 cpu_manager.go:414] "containerMap: removing container" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" Oct 30 21:37:48.979589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979589 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979603 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9e8beafa-7a51-4882-86cc-6e5e97e57953" containerName="discovery" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979611 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8beafa-7a51-4882-86cc-6e5e97e57953" containerName="discovery" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979631 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bb35621f-6627-4667-9b2b-b5e5adc7d654" containerName="maas-api" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979638 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb35621f-6627-4667-9b2b-b5e5adc7d654" containerName="maas-api" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979658 5039 cpu_manager.go:414] "containerMap: removing container" podUID="0d22a262-62f4-4a57-adab-9f7e5269c967" containerName="maas-api" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979665 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d22a262-62f4-4a57-adab-9f7e5269c967" containerName="maas-api" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979681 5039 cpu_manager.go:414] "containerMap: removing container" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979688 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979717 5039 cpu_manager.go:414] "containerMap: removing container" podUID="06dcdf53-89b6-4302-94ef-d59d05d1f718" containerName="maas-api" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979723 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dcdf53-89b6-4302-94ef-d59d05d1f718" containerName="maas-api" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979735 5039 cpu_manager.go:414] "containerMap: removing container" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" Oct 30 21:37:48.979769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979742 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979883 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979895 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8beafa-7a51-4882-86cc-6e5e97e57953" containerName="discovery" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979908 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb35621f-6627-4667-9b2b-b5e5adc7d654" containerName="maas-api" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979921 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d22a262-62f4-4a57-adab-9f7e5269c967" containerName="maas-api" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979930 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3e1041-a476-42d7-bb2c-d5d8b813161f" containerName="istio-proxy" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979941 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979951 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dcdf53-89b6-4302-94ef-d59d05d1f718" containerName="maas-api" Oct 30 21:37:48.980302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.979960 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="48af0e61-5d81-4ab2-b651-121bd5fff894" containerName="istio-proxy" Oct 30 21:37:48.985440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:48.985416 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.004141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.004123 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4e4693-openshift-default-dockercfg-bl2xx\"" Oct 30 21:37:49.013904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.013881 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx"] Oct 30 21:37:49.048262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048237 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048401 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048359 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048447 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rrj\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048513 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048583 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048639 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048717 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048763 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.048831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.048797 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.153056 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.153889 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.153954 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.153998 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154041 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154076 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154106 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154153 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154253 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154286 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rrj\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.156413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.154971 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.166740 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.166707 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.169578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.169555 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.176267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.171746 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.176267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.173304 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.176267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.175653 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.183697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.183673 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.183980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.183958 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.201134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.196335 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rrj\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj\") pod \"e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.299789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.299757 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:49.311058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.311020 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:49.614102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.614073 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx"] Oct 30 21:37:49.616165 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:37:49.616133 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c43ef7_f835_47a9_97fe_3381e96a1ea5.slice/crio-50bc991766c6b28656a9bdeb9588cdd709dcded52e7408ed1e00d9ad0ae14c67 WatchSource:0}: Error finding container 50bc991766c6b28656a9bdeb9588cdd709dcded52e7408ed1e00d9ad0ae14c67: Status 404 returned error can't find the container with id 50bc991766c6b28656a9bdeb9588cdd709dcded52e7408ed1e00d9ad0ae14c67 Oct 30 21:37:49.618995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.618955 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:49.619110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.619036 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:49.619110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:49.619082 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:50.180420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:50.180361 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" event={"ID":"26c43ef7-f835-47a9-97fe-3381e96a1ea5","Type":"ContainerStarted","Data":"6ae59c51f26fcda1a071c7b8aeade8f8802fd722e64bc2b540fc5f98888f1603"} Oct 30 21:37:50.180781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:50.180435 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" event={"ID":"26c43ef7-f835-47a9-97fe-3381e96a1ea5","Type":"ContainerStarted","Data":"50bc991766c6b28656a9bdeb9588cdd709dcded52e7408ed1e00d9ad0ae14c67"} Oct 30 21:37:50.238416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:50.238360 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podStartSLOduration=2.238348196 podStartE2EDuration="2.238348196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:50.226886878 +0000 UTC m=+1709.436826508" watchObservedRunningTime="2026-10-30 21:37:50.238348196 +0000 UTC m=+1709.448287824" Oct 30 21:37:50.300610 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:50.300586 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:50.305850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:50.305827 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:50.634088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:50.634011 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:51.185500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:51.185467 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:51.186431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:51.186380 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:37:52.431972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:52.431935 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:52.682452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:52.682355 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:52.844232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:52.844199 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:54.014150 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:54.014111 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:54.377555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:54.377474 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:54.557517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:54.557481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:54.713648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:54.713618 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:54.848049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:54.848015 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:55.183860 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:37:55.183828 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:37:55.289792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:55.289760 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:56.475947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:56.475907 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:56.972301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:56.971984 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.113162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:57.113128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.284416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:57.284172 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.672368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:57.672334 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.827766 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:57.827736 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.968716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:57.968635 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:58.329923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:58.329834 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:59.756335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:37:59.756297 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:01.042982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:01.042944 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:01.191917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:01.191884 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:01.395663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:01.395629 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:02.035844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:02.035787 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:03.630714 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:03.630673 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:04.220299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:04.220265 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:04.385751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:04.385707 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:05.093844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:05.093809 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:05.259027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:05.258990 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:05.429500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:05.429460 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:05.567485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:05.567457 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:38:05.594090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:05.594063 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:07.325738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:07.325688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:08.897639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:08.897607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:10.133936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:10.133898 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:10.426705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:10.426671 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:12.022029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:12.021997 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:12.166476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:12.166260 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:12.297470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:12.297373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:13.656654 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:13.656614 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:13.935630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:13.935511 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:15.409894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:15.409861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:15.550185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:15.550147 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:16.667410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:16.667351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:16.831576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:16.831546 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:17.107159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:17.107080 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:18.733916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:18.733721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:20.497547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:20.497510 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:21.030335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:21.030298 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:21.198693 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:21.198540 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:21.666926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:21.666894 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:22.908385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:22.908235 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:24.227853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:24.227802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:24.542890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:24.542792 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:24.801745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:24.801615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:24.943514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:24.943471 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:25.183574 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:38:25.183541 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:38:25.475868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:25.475780 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:25.737124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:25.737049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.693379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:26.693269 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:27.953528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:27.953496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:28.114742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:28.114700 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:28.295457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:28.295362 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:28.432918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:28.432886 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:29.265876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:29.265836 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:38:29.969097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:29.969058 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.707890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:31.707861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.878412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:31.878234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:32.078303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:32.078217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:32.298460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:32.294009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:32.512422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:32.512372 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:33.286516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:33.286484 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:33.809004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:33.808967 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:34.383572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:34.383534 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:34.544850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:34.544818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:35.875428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:35.869805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:36.048112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.048083 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t"] Oct 30 21:38:36.048647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.048632 5039 cpu_manager.go:414] "containerMap: removing container" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" Oct 30 21:38:36.048700 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.048648 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa063257-509c-4fb0-b726-9c9d5912f6f9" containerName="istio-proxy" Oct 30 21:38:36.053551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.053529 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.068669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.068647 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:36.078239 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.078218 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\"" Oct 30 21:38:36.083569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.083534 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t"] Oct 30 21:38:36.107379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.107352 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.107622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.107601 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.107749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.107725 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9f6\" (UniqueName: \"kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.208671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.208649 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.208777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.208706 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9f6\" (UniqueName: \"kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.208777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.208759 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.209384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.209362 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.211037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.211012 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.225478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.225441 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9f6\" (UniqueName: \"kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6\") pod \"maas-api-revoke-sub-71ef017f-g2s9t\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.368099 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.368069 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:36.676214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:36.676184 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t"] Oct 30 21:38:36.680296 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:38:36.680270 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc904c88_df88_4902_9d09_0581446bdfb7.slice/crio-7feb2cd0bc52631f15663562ddc42bd507be2261e8dcda204e011b39c09e09c1 WatchSource:0}: Error finding container 7feb2cd0bc52631f15663562ddc42bd507be2261e8dcda204e011b39c09e09c1: Status 404 returned error can't find the container with id 7feb2cd0bc52631f15663562ddc42bd507be2261e8dcda204e011b39c09e09c1 Oct 30 21:38:37.228607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:37.228528 5039 patch_prober.go:29] interesting pod/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:38:37.228607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:37.228579 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:38:37.510569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:37.510485 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" event={"ID":"dc904c88-df88-4902-9d09-0581446bdfb7","Type":"ContainerStarted","Data":"1b0a7d6f77b8869dac08ba169185f851e3edfdfb138dbb650709b5ba6f3ca523"} Oct 30 21:38:37.510569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:37.510542 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" event={"ID":"dc904c88-df88-4902-9d09-0581446bdfb7","Type":"ContainerStarted","Data":"7feb2cd0bc52631f15663562ddc42bd507be2261e8dcda204e011b39c09e09c1"} Oct 30 21:38:37.546674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:37.546624 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" podStartSLOduration=1.5466094479999999 podStartE2EDuration="1.546609448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:37.544024727 +0000 UTC m=+1756.753964355" watchObservedRunningTime="2026-10-30 21:38:37.546609448 +0000 UTC m=+1756.756549076" Oct 30 21:38:38.005642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:37.997193 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.519781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:38.519751 5039 generic.go:356] "Generic (PLEG): container finished" podID="dc904c88-df88-4902-9d09-0581446bdfb7" containerID="1b0a7d6f77b8869dac08ba169185f851e3edfdfb138dbb650709b5ba6f3ca523" exitCode=0 Oct 30 21:38:38.520157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:38.519835 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" event={"ID":"dc904c88-df88-4902-9d09-0581446bdfb7","Type":"ContainerDied","Data":"1b0a7d6f77b8869dac08ba169185f851e3edfdfb138dbb650709b5ba6f3ca523"} Oct 30 21:38:39.526972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.526915 5039 generic.go:356] "Generic (PLEG): container finished" podID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerID="6ae59c51f26fcda1a071c7b8aeade8f8802fd722e64bc2b540fc5f98888f1603" exitCode=0 Oct 30 21:38:39.527377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.526990 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" event={"ID":"26c43ef7-f835-47a9-97fe-3381e96a1ea5","Type":"ContainerDied","Data":"6ae59c51f26fcda1a071c7b8aeade8f8802fd722e64bc2b540fc5f98888f1603"} Oct 30 21:38:39.527797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.527558 5039 scope.go:122] "RemoveContainer" containerID="6ae59c51f26fcda1a071c7b8aeade8f8802fd722e64bc2b540fc5f98888f1603" Oct 30 21:38:39.528839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.528806 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:39.528923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.528874 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:39.528923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.528906 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:39.698177 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.698140 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:39.754187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.753836 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:39.851210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.851177 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca\") pod \"dc904c88-df88-4902-9d09-0581446bdfb7\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " Oct 30 21:38:39.851410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.851296 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca\") pod \"dc904c88-df88-4902-9d09-0581446bdfb7\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " Oct 30 21:38:39.851410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.851346 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6\" (UniqueName: \"kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6\") pod \"dc904c88-df88-4902-9d09-0581446bdfb7\" (UID: \"dc904c88-df88-4902-9d09-0581446bdfb7\") " Oct 30 21:38:39.851729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.851698 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca" pod "dc904c88-df88-4902-9d09-0581446bdfb7" (UID: "dc904c88-df88-4902-9d09-0581446bdfb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:39.853159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.853136 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca" pod "dc904c88-df88-4902-9d09-0581446bdfb7" (UID: "dc904c88-df88-4902-9d09-0581446bdfb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:39.853344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.853324 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6" pod "dc904c88-df88-4902-9d09-0581446bdfb7" (UID: "dc904c88-df88-4902-9d09-0581446bdfb7"). InnerVolumeSpecName "kube-api-access-vq9f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:39.952777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.952708 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-openshift-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:38:39.952777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.952741 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9f6\" (UniqueName: \"kubernetes.io/projected/dc904c88-df88-4902-9d09-0581446bdfb7-kube-api-access-vq9f6\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:38:39.952777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:39.952757 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc904c88-df88-4902-9d09-0581446bdfb7-maas-api-xks-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:38:40.533785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:40.533751 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" Oct 30 21:38:40.533785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:40.533765 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t" event={"ID":"dc904c88-df88-4902-9d09-0581446bdfb7","Type":"ContainerDied","Data":"7feb2cd0bc52631f15663562ddc42bd507be2261e8dcda204e011b39c09e09c1"} Oct 30 21:38:40.534242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:40.533808 5039 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7feb2cd0bc52631f15663562ddc42bd507be2261e8dcda204e011b39c09e09c1" Oct 30 21:38:40.535842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:40.535801 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" event={"ID":"26c43ef7-f835-47a9-97fe-3381e96a1ea5","Type":"ContainerStarted","Data":"2cfb89dd075e50069c7e877535b67060cf0e4d33e9b20cd6ec95a9aefb0daff0"} Oct 30 21:38:41.221322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:41.220348 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:41.300957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:41.300926 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:38:41.301440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:41.301415 5039 patch_prober.go:29] interesting pod/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:38:41.301561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:41.301473 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:38:42.300108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:42.300075 5039 patch_prober.go:29] interesting pod/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:38:42.300514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:42.300131 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:38:42.883765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:42.883707 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:43.300477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:43.300443 5039 patch_prober.go:29] interesting pod/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:38:43.300880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:43.300497 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:38:44.300950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:44.300920 5039 patch_prober.go:29] interesting pod/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:38:44.301316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:44.300970 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 21:38:44.576234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:44.576124 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:45.304146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:45.304117 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:38:45.304631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:45.304610 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:38:45.305493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:45.305472 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:38:46.108636 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:46.108600 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:46.269281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:46.269243 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:47.825724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:47.825690 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:49.399836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:49.399787 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:50.729217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:50.729180 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:50.890561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:50.890529 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.042518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.042440 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.190940 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.190902 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.342843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.342737 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.381708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.381682 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:38:51.645913 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.645882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.784056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.783998 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:51.924372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:51.924299 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:52.099069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:52.096161 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:52.286607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:52.286523 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:53.911056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:53.911018 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:55.183161 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:38:55.183128 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:38:55.589626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:55.589551 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.178307 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:57.178265 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.657100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:57.657067 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf"] Oct 30 21:38:57.657282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:57.657262 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" containerID="cri-o://df20040c1260416969cb9eb88b1842ef3a0f4231cbff20f5d0b28196a60771ea" gracePeriod=30 Oct 30 21:38:58.232134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:58.232098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:58.386511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:58.386476 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:58.676142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:58.674614 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:59.034676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:59.034594 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:59.281192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:59.281153 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:59.443529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:59.443486 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:59.622680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:38:59.622639 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:00.223278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:00.223226 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:00.706512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:00.706478 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:00.886414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:00.883157 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:01.031929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:01.031834 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:02.775120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.775069 5039 generic.go:356] "Generic (PLEG): container finished" podID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerID="df20040c1260416969cb9eb88b1842ef3a0f4231cbff20f5d0b28196a60771ea" exitCode=0 Oct 30 21:39:02.775607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.775140 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" event={"ID":"a7214320-9109-4afe-ab03-0ae40f5861ea","Type":"ContainerDied","Data":"df20040c1260416969cb9eb88b1842ef3a0f4231cbff20f5d0b28196a60771ea"} Oct 30 21:39:02.842426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.839738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:02.938250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.938188 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:39:02.985907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.985883 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.985953 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.985991 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986020 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986099 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986132 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986178 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986229 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.986367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986294 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data\") pod \"a7214320-9109-4afe-ab03-0ae40f5861ea\" (UID: \"a7214320-9109-4afe-ab03-0ae40f5861ea\") " Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986592 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986601 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986926 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986941 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986941 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.986972 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:02.987258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.987230 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:02.988467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.988438 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:39:02.988772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.988747 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:02.989155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.989127 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:02.989540 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:02.989522 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27" pod "a7214320-9109-4afe-ab03-0ae40f5861ea" (UID: "a7214320-9109-4afe-ab03-0ae40f5861ea"). InnerVolumeSpecName "kube-api-access-xjj27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:03.088621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088598 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.088621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088623 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjj27\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-kube-api-access-xjj27\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.088751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088632 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.088751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088640 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.088751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088649 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7214320-9109-4afe-ab03-0ae40f5861ea-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.088751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088656 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7214320-9109-4afe-ab03-0ae40f5861ea-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.088751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.088664 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7214320-9109-4afe-ab03-0ae40f5861ea-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:03.782815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.782781 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" Oct 30 21:39:03.783189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.782821 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf" event={"ID":"a7214320-9109-4afe-ab03-0ae40f5861ea","Type":"ContainerDied","Data":"474dfbb60c804ea0e01982de753146823d5f78296cef556e45c30aafc1fa4f27"} Oct 30 21:39:03.783189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.782879 5039 scope.go:122] "RemoveContainer" containerID="df20040c1260416969cb9eb88b1842ef3a0f4231cbff20f5d0b28196a60771ea" Oct 30 21:39:03.837359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.837336 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf"] Oct 30 21:39:03.846814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:03.846792 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lbmvf"] Oct 30 21:39:04.682326 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:04.682283 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:04.922644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:04.922606 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" path="/var/lib/kubelet/pods/a7214320-9109-4afe-ab03-0ae40f5861ea/volumes" Oct 30 21:39:04.929891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:04.929851 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:05.204437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:05.204371 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:06.699888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:06.699855 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:07.489023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:07.488994 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:07.637784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:07.637746 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:08.379522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:08.379482 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:10.092877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:10.092625 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:11.575142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:11.575101 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:12.695648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.695616 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw"] Oct 30 21:39:12.696108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.696092 5039 cpu_manager.go:414] "containerMap: removing container" podUID="dc904c88-df88-4902-9d09-0581446bdfb7" containerName="revoke-keys" Oct 30 21:39:12.696157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.696110 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc904c88-df88-4902-9d09-0581446bdfb7" containerName="revoke-keys" Oct 30 21:39:12.696157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.696141 5039 cpu_manager.go:414] "containerMap: removing container" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" Oct 30 21:39:12.696157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.696146 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" Oct 30 21:39:12.696248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.696231 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc904c88-df88-4902-9d09-0581446bdfb7" containerName="revoke-keys" Oct 30 21:39:12.696248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.696245 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7214320-9109-4afe-ab03-0ae40f5861ea" containerName="istio-proxy" Oct 30 21:39:12.701067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.701047 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.728085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.728060 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw"] Oct 30 21:39:12.796801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.796770 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.796938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.796841 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.796938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.796897 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbsh\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.796938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.796931 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.797049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.797036 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.797088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.797071 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.797120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.797092 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.797201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.797183 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.797260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.797247 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898348 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898382 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898419 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898464 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898480 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898626 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898660 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbsh\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898694 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898770 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898783 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.898883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.898789 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.899044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.899007 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.899044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.899022 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.899327 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.899307 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.900698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.900676 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.900928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.900909 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.915816 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.915788 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:12.916017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:12.915996 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbsh\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:13.014184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.014133 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:13.197447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.197384 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:13.318638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.318610 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw"] Oct 30 21:39:13.319218 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:39:13.319188 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192f7042_ada8_40d3_b5b1_5be2f8731c70.slice/crio-2e317ea203fb9765b177c4e60aa4da8f478468795729acb084096352369b8775 WatchSource:0}: Error finding container 2e317ea203fb9765b177c4e60aa4da8f478468795729acb084096352369b8775: Status 404 returned error can't find the container with id 2e317ea203fb9765b177c4e60aa4da8f478468795729acb084096352369b8775 Oct 30 21:39:13.321578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.321542 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:13.321660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.321609 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:13.321660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.321638 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:13.740865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.740825 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:13.852451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.852418 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" event={"ID":"192f7042-ada8-40d3-b5b1-5be2f8731c70","Type":"ContainerStarted","Data":"5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d"} Oct 30 21:39:13.852612 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.852502 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" event={"ID":"192f7042-ada8-40d3-b5b1-5be2f8731c70","Type":"ContainerStarted","Data":"2e317ea203fb9765b177c4e60aa4da8f478468795729acb084096352369b8775"} Oct 30 21:39:13.908523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:13.908466 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podStartSLOduration=1.908450635 podStartE2EDuration="1.908450635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:13.907209472 +0000 UTC m=+1793.117149100" watchObservedRunningTime="2026-10-30 21:39:13.908450635 +0000 UTC m=+1793.118390263" Oct 30 21:39:14.014367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:14.014287 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:14.015593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:14.015569 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:39:14.015711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:14.015621 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:39:15.002356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:15.002206 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:15.021139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:15.021100 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:39:15.021313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:15.021165 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:39:16.018486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:16.018450 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:39:16.019139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:16.019107 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:39:16.294421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:16.294294 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.440318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:16.440237 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.012790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:17.012756 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.016060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:17.015262 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:39:17.016060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:17.015350 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:39:17.151855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:17.151822 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.285847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:17.285768 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:18.014997 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:18.014951 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:39:18.015176 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:18.015017 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:39:19.014453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:19.014405 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:39:19.014960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:19.014483 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:39:19.431190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:19.431160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:21.055516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.055484 5039 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:21.055870 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.055527 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:21.165685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.165654 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:39:21.174530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.174500 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:39:21.313133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.313065 5039 scope.go:122] "RemoveContainer" containerID="3d9e39cfd345c611e0827799383a1f871ead6c89705e8a32dd32b7e1f57dd1fb" Oct 30 21:39:21.339236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.339196 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:21.339547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.339484 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:21.340676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.340660 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:39:21.820333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:21.820297 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:24.195683 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:24.195648 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:25.183769 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:39:25.183736 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:39:25.689909 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:25.689864 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:26.229906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:26.229868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:26.469347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:26.469302 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:26.625050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:26.624971 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:26.989651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:26.989612 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:28.509193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:28.509159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:30.105480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:30.105450 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.802925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:31.802895 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:32.881529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:32.881497 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc"] Oct 30 21:39:32.886992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:32.886968 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:32.911349 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:32.911323 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc"] Oct 30 21:39:33.001854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.001828 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.001985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.001879 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.001985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.001898 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.002073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.002053 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.002112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.002093 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.002167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.002155 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.002203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.002181 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.002236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.002202 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.002236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.002223 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5g2\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105185 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105258 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105315 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105352 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105381 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105429 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5g2\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105523 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105593 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.105621 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.106360 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.106988 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.107218 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.121302 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.121627 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.137468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.130879 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.144426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.144375 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.147571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.147536 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5g2\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.151691 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.151641 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.202990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.202959 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:33.302264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.302207 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.451464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.451364 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.517252 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.517228 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc"] Oct 30 21:39:33.518404 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:39:33.518370 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303fe40a_e370_4a57_943f_a2be80c97f30.slice/crio-022b974b4a9daf5c20997a1e07ed078adf8bd241f499f6e5de1afefbfa190396 WatchSource:0}: Error finding container 022b974b4a9daf5c20997a1e07ed078adf8bd241f499f6e5de1afefbfa190396: Status 404 returned error can't find the container with id 022b974b4a9daf5c20997a1e07ed078adf8bd241f499f6e5de1afefbfa190396 Oct 30 21:39:33.520727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.520694 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:33.520810 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.520765 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:33.520810 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.520794 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:33.584456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.578497 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx"] Oct 30 21:39:33.584456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.578786 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" containerID="cri-o://2cfb89dd075e50069c7e877535b67060cf0e4d33e9b20cd6ec95a9aefb0daff0" gracePeriod=30 Oct 30 21:39:33.600503 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.600362 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.786897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:33.786805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.150597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:34.150568 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" event={"ID":"303fe40a-e370-4a57-943f-a2be80c97f30","Type":"ContainerStarted","Data":"75451fe2e11d20f17e7a56801d9202838ec1843d879ec90fd1088882ccb8696d"} Oct 30 21:39:34.150941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:34.150618 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" event={"ID":"303fe40a-e370-4a57-943f-a2be80c97f30","Type":"ContainerStarted","Data":"022b974b4a9daf5c20997a1e07ed078adf8bd241f499f6e5de1afefbfa190396"} Oct 30 21:39:34.203453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:34.203427 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:34.205006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:34.204984 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:34.205060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:34.205033 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:34.208528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:34.208490 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podStartSLOduration=2.208477924 podStartE2EDuration="2.208477924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.196843856 +0000 UTC m=+1813.406783481" watchObservedRunningTime="2026-10-30 21:39:34.208477924 +0000 UTC m=+1813.418417553" Oct 30 21:39:35.183969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:35.183437 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.204248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:35.204211 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:35.204488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:35.204268 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:35.399030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:35.398996 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:36.203635 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:36.203588 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:36.204031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:36.203655 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:36.417991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:36.417959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:36.772307 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:36.772276 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:39:37.026609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:37.026519 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:37.186514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:37.186480 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:37.204687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:37.204655 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:37.205132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:37.205108 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:37.341354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:37.341264 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.204060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:38.204024 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:38.204364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:38.204321 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:39.203449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:39.203336 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:39.204065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:39.204022 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:39.254277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:39.254243 5039 generic.go:356] "Generic (PLEG): container finished" podID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerID="2cfb89dd075e50069c7e877535b67060cf0e4d33e9b20cd6ec95a9aefb0daff0" exitCode=0 Oct 30 21:39:39.254669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:39.254646 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" event={"ID":"26c43ef7-f835-47a9-97fe-3381e96a1ea5","Type":"ContainerDied","Data":"2cfb89dd075e50069c7e877535b67060cf0e4d33e9b20cd6ec95a9aefb0daff0"} Oct 30 21:39:39.254833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:39.254820 5039 scope.go:122] "RemoveContainer" containerID="6ae59c51f26fcda1a071c7b8aeade8f8802fd722e64bc2b540fc5f98888f1603" Oct 30 21:39:40.130451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.130415 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:39:40.228059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.228025 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.230011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.229984 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:40.230200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.230172 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:40.268468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.265779 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.268468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.265833 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.268468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.265965 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.268468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.266032 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.288491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.284685 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:40.288491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.287583 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:40.291529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.291481 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:40.310321 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.310294 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.312027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.312003 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.325056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.324994 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.325056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.325040 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.325263 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.325084 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token\") pod \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\" (UID: \"26c43ef7-f835-47a9-97fe-3381e96a1ea5\") " Oct 30 21:39:40.325750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.325666 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.325750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.325692 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.325750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.325706 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.325960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.312376 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:40.326012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.312496 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:40.327861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.327835 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" event={"ID":"26c43ef7-f835-47a9-97fe-3381e96a1ea5","Type":"ContainerDied","Data":"50bc991766c6b28656a9bdeb9588cdd709dcded52e7408ed1e00d9ad0ae14c67"} Oct 30 21:39:40.328021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.328009 5039 scope.go:122] "RemoveContainer" containerID="2cfb89dd075e50069c7e877535b67060cf0e4d33e9b20cd6ec95a9aefb0daff0" Oct 30 21:39:40.328206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.328195 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx" Oct 30 21:39:40.353259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.353232 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:40.354254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.354230 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:40.354823 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.354788 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "kube-api-access-k4rrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:40.366040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.366001 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo" pod "26c43ef7-f835-47a9-97fe-3381e96a1ea5" (UID: "26c43ef7-f835-47a9-97fe-3381e96a1ea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:39:40.434448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.428492 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.434448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.428533 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.434448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.428551 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.434448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.428565 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.434448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.428580 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.434448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.428594 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rrj\" (UniqueName: \"kubernetes.io/projected/26c43ef7-f835-47a9-97fe-3381e96a1ea5-kube-api-access-k4rrj\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:39:40.732443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.731441 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx"] Oct 30 21:39:40.774414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.773672 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e4693-openshift-default-869bcd75f7-zgrjx"] Oct 30 21:39:40.931182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:40.931091 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" path="/var/lib/kubelet/pods/26c43ef7-f835-47a9-97fe-3381e96a1ea5/volumes" Oct 30 21:39:41.204147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:41.204064 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:39:41.204279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:41.204123 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:39:42.491649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:42.491612 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.691046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:42.691009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.908164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:42.908132 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:42.908614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:42.908592 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:42.909456 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:42.909434 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:39:44.335503 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:44.335467 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:44.513252 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:44.513214 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:46.263984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:46.263950 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:48.013366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:48.013328 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:49.715197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:49.715162 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.208167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:51.208127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.348706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:51.348669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.488615 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:51.488547 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:53.308057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:53.308025 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.812584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:54.812551 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.957128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:54.957091 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:55.184492 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:39:55.184454 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:39:57.487414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:57.483825 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:57.736516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:57.736470 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 21:39:57.796584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:57.796478 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.205642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:39:58.205566 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.504170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:00.504142 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:02.385694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.385650 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:02.973563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.973535 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm"] Oct 30 21:40:02.974132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.974112 5039 cpu_manager.go:414] "containerMap: removing container" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" Oct 30 21:40:02.974132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.974131 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" Oct 30 21:40:02.974275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.974140 5039 cpu_manager.go:414] "containerMap: removing container" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" Oct 30 21:40:02.974275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.974145 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" Oct 30 21:40:02.974275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.974242 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" Oct 30 21:40:02.974275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.974252 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c43ef7-f835-47a9-97fe-3381e96a1ea5" containerName="istio-proxy" Oct 30 21:40:02.977838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:02.977818 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.004781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.004754 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm"] Oct 30 21:40:03.078032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078000 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078042 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078068 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078161 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078193 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078274 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078223 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7xk\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078340 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078376 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.078493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.078476 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179246 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179273 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179290 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179310 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179328 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179357 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7xk\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179442 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179482 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179531 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.179829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.179688 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.180101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.180054 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.180101 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.180068 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.180206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.180125 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.180258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.180240 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.181559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.181536 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.181787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.181768 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.195854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.195831 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.195956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.195908 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7xk\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.291263 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.291209 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:03.600933 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.600905 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm"] Oct 30 21:40:03.601256 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:03.601037 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1b2d63_7d92_4eac_9d71_46038b002151.slice/crio-bdb4f48a056bdf87c42ae66cbab7bec61774dd8e42748f9422833f5965948348 WatchSource:0}: Error finding container bdb4f48a056bdf87c42ae66cbab7bec61774dd8e42748f9422833f5965948348: Status 404 returned error can't find the container with id bdb4f48a056bdf87c42ae66cbab7bec61774dd8e42748f9422833f5965948348 Oct 30 21:40:03.603417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.603363 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:03.603517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.603465 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:03.603517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.603504 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:03.884056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:03.883915 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.505469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:04.505427 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" event={"ID":"8a1b2d63-7d92-4eac-9d71-46038b002151","Type":"ContainerStarted","Data":"11684b25b18ae0e1cd7e5009064984fd859262a499019bf40eba80ad71d92bbc"} Oct 30 21:40:04.505650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:04.505505 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" event={"ID":"8a1b2d63-7d92-4eac-9d71-46038b002151","Type":"ContainerStarted","Data":"bdb4f48a056bdf87c42ae66cbab7bec61774dd8e42748f9422833f5965948348"} Oct 30 21:40:04.577412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:04.557636 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podStartSLOduration=2.557620637 podStartE2EDuration="2.557620637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:04.555023994 +0000 UTC m=+1843.764963623" watchObservedRunningTime="2026-10-30 21:40:04.557620637 +0000 UTC m=+1843.767560266" Oct 30 21:40:05.291690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:05.291651 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:05.292992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:05.292963 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:40:05.293133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:05.293028 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:40:05.607706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:05.607616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:06.292038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:06.292007 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:40:06.292523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:06.292062 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:40:07.292310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:07.292273 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:40:07.292746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:07.292334 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:40:07.408614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:07.408579 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:07.582469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:07.582067 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:07.905730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:07.905698 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.074591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:08.074557 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.292618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:08.292533 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:40:08.292618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:08.292596 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:40:09.227747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:09.227696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:09.291553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:09.291520 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:40:09.291738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:09.291571 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:40:10.382178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:10.382145 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:10.382716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:10.382692 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:10.383635 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:10.383615 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:40:10.900840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:10.900800 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.463681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:11.463644 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.628916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:11.628869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.792132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:11.792056 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:12.382659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:12.382629 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:40:12.436143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:12.436108 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:12.610433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:12.610343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:13.398239 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:13.398208 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:13.595953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:13.595913 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.099413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:14.096920 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.747529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:14.747478 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:14.855534 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:14.855485 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:14.972424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:14.958759 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:14.999192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:14.999092 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.007015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.001881 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 21:40:15.120416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.117479 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.223234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.200762 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.296322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.296216 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.478118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.477963 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.644902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.638208 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.749550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.749500 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.794529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.794481 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.858113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.858055 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:15.874421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.867458 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb"] Oct 30 21:40:15.933465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.913761 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb"] Oct 30 21:40:15.933465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.913855 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:40:15.933465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.913946 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:40:15.933465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:15.914731 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.004489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.004452 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.015865 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017320 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017426 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017457 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017495 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jp2m\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017610 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017664 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017835 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017888 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.060767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.017930 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.125414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.092576 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:16.164413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.131435 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:16.164413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.157167 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:40:16.165473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.165447 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.165991 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.166196 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166360 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.166344 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.166502 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.166646 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.166804 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.166932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.166915 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.190452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.168062 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jp2m\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.193004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.192975 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.193829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.193807 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.212275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.194466 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 21:40:16.212491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.194739 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.212609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.195185 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.212703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.167495 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.212802 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.212136 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:16.214507 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.213735 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:40:16.259621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.254619 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.259621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.256828 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.270558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.270533 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jp2m\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.325449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.318240 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 21:40:16.360434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.348796 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.573880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.573799 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:16.714377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.714336 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 21:40:16.819411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.816812 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:16.820750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.820714 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:40:16.862374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:16.861593 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 21:40:17.177420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.166057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.179047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.179011 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:17.241531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.241478 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:17.242291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.242265 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:40:17.255527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.255483 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:17.360414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.335792 5039 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:17.503818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.484697 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.879465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.800788 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb"] Oct 30 21:40:17.924055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:17.923447 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:40:18.011406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.011355 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:18.016445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.011665 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:18.022427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.018884 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:18.025371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.025349 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 21:40:18.037799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.037779 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 21:40:18.073090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.073060 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" event={"ID":"65e6f922-b778-47b5-ad07-62f41980f30f","Type":"ContainerStarted","Data":"21b53beeef3a155cf4570a52712af627e87c69f9a84ff1ff753e48fe0ff64dde"} Oct 30 21:40:18.095460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.095432 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l"] Oct 30 21:40:18.124936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.124916 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.141648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.126976 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 21:40:18.147669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.143967 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l"] Oct 30 21:40:18.237584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237461 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6zf\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.237584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237529 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237609 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237748 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237805 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237889 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.237977 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.238135 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.238187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.238177 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.339528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.339496 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.339733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.339541 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.339733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.339575 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6zf\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.339733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.339593 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.339733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.339644 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.339999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.339967 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340035 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340008 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340035 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340008 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340053 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340089 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340163 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340303 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340284 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340342 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.340554 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.340532 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.342302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.342275 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.342539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.342518 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.356696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.356674 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.356922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.356903 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6zf\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.516994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.516900 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:18.837060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.837033 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l"] Oct 30 21:40:18.837738 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:18.837709 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e92786_2a34_4fc8_8f8d_da19ee087997.slice/crio-0c8459ac772281fcc129b8f60526b3071b95dbb535765ed60b9ed82ca4617a0e WatchSource:0}: Error finding container 0c8459ac772281fcc129b8f60526b3071b95dbb535765ed60b9ed82ca4617a0e: Status 404 returned error can't find the container with id 0c8459ac772281fcc129b8f60526b3071b95dbb535765ed60b9ed82ca4617a0e Oct 30 21:40:18.840164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.840132 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:18.840255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.840195 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:18.840255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:18.840224 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:19.093204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.093115 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" event={"ID":"51e92786-2a34-4fc8-8f8d-da19ee087997","Type":"ContainerStarted","Data":"7c0b415c0271672a13ac9b33cea2afa91a2673182ccaeeb170890f2db6125e05"} Oct 30 21:40:19.093204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.093167 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" event={"ID":"51e92786-2a34-4fc8-8f8d-da19ee087997","Type":"ContainerStarted","Data":"0c8459ac772281fcc129b8f60526b3071b95dbb535765ed60b9ed82ca4617a0e"} Oct 30 21:40:19.094878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.094848 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" event={"ID":"65e6f922-b778-47b5-ad07-62f41980f30f","Type":"ContainerStarted","Data":"078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec"} Oct 30 21:40:19.132304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.132257 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podStartSLOduration=1.132244029 podStartE2EDuration="1.132244029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.131259577 +0000 UTC m=+1858.341199206" watchObservedRunningTime="2026-10-30 21:40:19.132244029 +0000 UTC m=+1858.342183657" Oct 30 21:40:19.321023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.320988 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:19.519487 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.519381 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:19.520067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.520040 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:19.520232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.520207 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:19.587481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.587447 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:19.624404 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.624347 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:40:19.624575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.624440 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:40:19.696415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.694769 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:19.931695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:19.931648 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:20.517297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:20.517269 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:20.517758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:20.517733 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:20.578412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:20.577475 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:40:20.578412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:20.577526 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:40:21.517915 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:21.517882 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:21.518252 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:21.517934 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:21.579412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:21.579204 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:40:21.579412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:21.579256 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:40:22.187418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.186507 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:22.448692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.448608 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:22.517877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.517843 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:22.518043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.517895 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:22.583415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.576760 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:40:22.583415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.576817 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:40:22.783464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:22.782981 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.517993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:23.517958 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:23.518194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:23.518015 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:23.580420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:23.578741 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:40:23.580420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:23.578803 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:40:24.303492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.303459 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:24.488770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.488734 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:24.518047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.517944 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:24.518047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.518003 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:24.584896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.584814 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:40:24.585165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.585136 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:40:24.689333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:24.689257 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:25.194650 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:40:25.194610 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:40:25.517890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:25.517809 5039 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:40:25.517890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:25.517865 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:40:25.579409 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:25.579364 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:25.579820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:25.579664 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:25.580658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:25.580637 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:40:25.628753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:25.628712 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podStartSLOduration=10.628699561 podStartE2EDuration="10.628699561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.189709817 +0000 UTC m=+1858.399649449" watchObservedRunningTime="2026-10-30 21:40:25.628699561 +0000 UTC m=+1864.838639189" Oct 30 21:40:26.521126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:26.521093 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:26.521568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:26.521547 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:26.522092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:26.522074 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:40:26.809959 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:26.809885 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:27.478193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:27.478163 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5"] Oct 30 21:40:27.478377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:27.478348 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" containerID="cri-o://a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b" gracePeriod=30 Oct 30 21:40:27.992505 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:27.992480 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs"] Oct 30 21:40:27.992950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:27.992656 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" containerID="cri-o://be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d" gracePeriod=30 Oct 30 21:40:28.816245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:28.816213 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:30.225531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.223521 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:30.249413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.241471 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:30.303538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.303492 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:30.325572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.325535 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:30.892347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.892312 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6"] Oct 30 21:40:30.906147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.906125 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:30.926527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.926501 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c19b75\"/\"kube-root-ca.crt\"" Oct 30 21:40:30.926859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.926839 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c19b75\"/\"openshift-service-ca.crt\"" Oct 30 21:40:30.945927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.938523 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c19b75\"/\"praxis-route-c19b75-kserve-self-signed-certs\"" Oct 30 21:40:30.945927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.938580 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c19b75\"/\"default-dockercfg-hlpld\"" Oct 30 21:40:30.948439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:30.948420 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6"] Oct 30 21:40:31.002526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.002488 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:31.039225 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.039195 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.039521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.039502 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.039766 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.039746 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.039922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.039907 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.040093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.040076 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dml8h\" (UniqueName: \"kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.040337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.040299 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.141904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.141860 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.141948 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142037 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142083 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142186 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142156 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dml8h\" (UniqueName: \"kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142202 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142411 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142500 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.142674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.142594 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.144543 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.144521 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.144655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.144576 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.161979 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.161931 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dml8h\" (UniqueName: \"kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h\") pod \"praxis-route-c19b75-kserve-64dc9c7895-ftzb6\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.246608 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.246570 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:31.587848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:31.587821 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6"] Oct 30 21:40:31.588814 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:31.588791 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6969b10c_c154_4cd5_a778_f7d7c6a3549b.slice/crio-5470250f458c972320101e83c407170105011b03555550759d0aaed33d406c70 WatchSource:0}: Error finding container 5470250f458c972320101e83c407170105011b03555550759d0aaed33d406c70: Status 404 returned error can't find the container with id 5470250f458c972320101e83c407170105011b03555550759d0aaed33d406c70 Oct 30 21:40:32.394124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.393939 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" event={"ID":"6969b10c-c154-4cd5-a778-f7d7c6a3549b","Type":"ContainerStarted","Data":"b3b3086572f0f2ac4df701f74994641c20f3b4aec33a07a0b943dbd44bf4ce9b"} Oct 30 21:40:32.394531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.394147 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" event={"ID":"6969b10c-c154-4cd5-a778-f7d7c6a3549b","Type":"ContainerStarted","Data":"5470250f458c972320101e83c407170105011b03555550759d0aaed33d406c70"} Oct 30 21:40:32.433011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.432951 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" podStartSLOduration=2.432934232 podStartE2EDuration="2.432934232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:32.42902005 +0000 UTC m=+1871.638959678" watchObservedRunningTime="2026-10-30 21:40:32.432934232 +0000 UTC m=+1871.642873860" Oct 30 21:40:32.433679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.433653 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:32.773016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.772995 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:40:32.867568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867535 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867627 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867664 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867707 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867689 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867733 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867762 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867824 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.867898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867876 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.868094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.867944 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs\") pod \"47e7b98a-0189-4670-901b-1fa3753a86fc\" (UID: \"47e7b98a-0189-4670-901b-1fa3753a86fc\") " Oct 30 21:40:32.868187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868158 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:32.868240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868113 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:32.868359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868318 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:32.868427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868379 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:32.868643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868621 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:32.868775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868688 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e7b98a-0189-4670-901b-1fa3753a86fc-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.868900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868776 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.868900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868794 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.868900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.868810 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.869871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.869840 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:32.870232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.870211 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:32.870620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.870595 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:40:32.870620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.870615 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c" pod "47e7b98a-0189-4670-901b-1fa3753a86fc" (UID: "47e7b98a-0189-4670-901b-1fa3753a86fc"). InnerVolumeSpecName "kube-api-access-ntd9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:32.943844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.943816 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-26mhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:40:32.943955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.943875 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:40:32.969945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.969922 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.969945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.969946 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.970086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.969955 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.970086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.969963 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntd9c\" (UniqueName: \"kubernetes.io/projected/47e7b98a-0189-4670-901b-1fa3753a86fc-kube-api-access-ntd9c\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:32.970086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:32.969973 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e7b98a-0189-4670-901b-1fa3753a86fc-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.248277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.248253 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:33.257498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.257466 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:40:33.379297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379269 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379309 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379406 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379454 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379503 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379528 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379552 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:33.379632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379563 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379670 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379707 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9\") pod \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\" (UID: \"6a24aabb-824b-4ba2-bfee-aaaae6bd997f\") " Oct 30 21:40:33.379849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379762 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:33.380008 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379873 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:33.380008 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.379895 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:33.380008 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.380000 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:33.380314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.380293 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.380386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.380319 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.380386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.380333 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.380386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.380348 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.380386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.380360 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.381729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.381703 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:33.381832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.381756 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:40:33.381832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.381774 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:33.381832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.381809 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9" pod "6a24aabb-824b-4ba2-bfee-aaaae6bd997f" (UID: "6a24aabb-824b-4ba2-bfee-aaaae6bd997f"). InnerVolumeSpecName "kube-api-access-rzpr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:33.401549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.401526 5039 generic.go:356] "Generic (PLEG): container finished" podID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerID="be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d" exitCode=0 Oct 30 21:40:33.401891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.401565 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" Oct 30 21:40:33.401891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.401606 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" event={"ID":"6a24aabb-824b-4ba2-bfee-aaaae6bd997f","Type":"ContainerDied","Data":"be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d"} Oct 30 21:40:33.401891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.401659 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs" event={"ID":"6a24aabb-824b-4ba2-bfee-aaaae6bd997f","Type":"ContainerDied","Data":"8282d3fb6070b716e0765835b1327e9bc75d4a6b4c57008475390196b1360744"} Oct 30 21:40:33.401891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.401683 5039 scope.go:122] "RemoveContainer" containerID="be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d" Oct 30 21:40:33.403160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.403134 5039 generic.go:356] "Generic (PLEG): container finished" podID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerID="a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b" exitCode=0 Oct 30 21:40:33.403268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.403181 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" Oct 30 21:40:33.403268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.403212 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" event={"ID":"47e7b98a-0189-4670-901b-1fa3753a86fc","Type":"ContainerDied","Data":"a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b"} Oct 30 21:40:33.403268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.403243 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5" event={"ID":"47e7b98a-0189-4670-901b-1fa3753a86fc","Type":"ContainerDied","Data":"7d6287c6c1ee97d021f7c89ca42259363f5e75dc75cc72abad0f58a491c09678"} Oct 30 21:40:33.410964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.410944 5039 scope.go:122] "RemoveContainer" containerID="be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d" Oct 30 21:40:33.411209 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:40:33.411193 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d\": container with ID starting with be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d not found: ID does not exist" containerID="be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d" Oct 30 21:40:33.411250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.411219 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d"} err="failed to get container status \"be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d\": rpc error: code = NotFound desc = could not find container \"be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d\": container with ID starting with be5cf72afaa997d7a6e0b3befd2d1c3a0b1868c3d2d127d501a352e5d597e34d not found: ID does not exist" Oct 30 21:40:33.411250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.411233 5039 scope.go:122] "RemoveContainer" containerID="a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b" Oct 30 21:40:33.419377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.419358 5039 scope.go:122] "RemoveContainer" containerID="a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b" Oct 30 21:40:33.419698 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:40:33.419678 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b\": container with ID starting with a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b not found: ID does not exist" containerID="a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b" Oct 30 21:40:33.419776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.419706 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b"} err="failed to get container status \"a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b\": rpc error: code = NotFound desc = could not find container \"a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b\": container with ID starting with a0b8bf5aa80f5f547bb813de532583b4644b3d23c1c75cb4ccfb0a902020741b not found: ID does not exist" Oct 30 21:40:33.450353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.450322 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs"] Oct 30 21:40:33.454109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.454086 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-26mhs"] Oct 30 21:40:33.481405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.481324 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.481405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.481351 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.481405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.481365 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.481405 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.481378 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpr9\" (UniqueName: \"kubernetes.io/projected/6a24aabb-824b-4ba2-bfee-aaaae6bd997f-kube-api-access-rzpr9\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:40:33.482361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.482340 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5"] Oct 30 21:40:33.486210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:33.486189 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pm6j5"] Oct 30 21:40:34.313538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.313505 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:34.344613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.344585 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz"] Oct 30 21:40:34.345073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.345058 5039 cpu_manager.go:414] "containerMap: removing container" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" Oct 30 21:40:34.345073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.345074 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" Oct 30 21:40:34.345182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.345102 5039 cpu_manager.go:414] "containerMap: removing container" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" Oct 30 21:40:34.345182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.345107 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" Oct 30 21:40:34.345246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.345203 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" containerName="istio-proxy" Oct 30 21:40:34.345246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.345216 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" containerName="istio-proxy" Oct 30 21:40:34.349876 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.349856 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.362366 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.362351 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-09eabe-serving-cert\"" Oct 30 21:40:34.376030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.376008 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz"] Oct 30 21:40:34.389290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.389264 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.389458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.389437 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.389529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.389467 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.389602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.389589 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.488017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.487987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:34.491157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.491133 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.491276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.491162 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.491276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.491234 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.491482 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.491316 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.491980 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.491956 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.493561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.493537 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.493712 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.493692 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.508146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.508120 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8\") pod \"maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.648371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.648332 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:34.662180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.662151 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:34.923839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.923764 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e7b98a-0189-4670-901b-1fa3753a86fc" path="/var/lib/kubelet/pods/47e7b98a-0189-4670-901b-1fa3753a86fc/volumes" Oct 30 21:40:34.924249 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.924236 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a24aabb-824b-4ba2-bfee-aaaae6bd997f" path="/var/lib/kubelet/pods/6a24aabb-824b-4ba2-bfee-aaaae6bd997f/volumes" Oct 30 21:40:34.960687 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:34.960278 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9761f2_8a71_4202_881a_13c7572defdd.slice/crio-ead51785cfbaf34c3419d63055408a00d76f534d9fa05343113841766c036eda WatchSource:0}: Error finding container ead51785cfbaf34c3419d63055408a00d76f534d9fa05343113841766c036eda: Status 404 returned error can't find the container with id ead51785cfbaf34c3419d63055408a00d76f534d9fa05343113841766c036eda Oct 30 21:40:34.962479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.962452 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz"] Oct 30 21:40:34.964294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:34.964273 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:40:35.421350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.421262 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" event={"ID":"ea9761f2-8a71-4202-881a-13c7572defdd","Type":"ContainerStarted","Data":"52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638"} Oct 30 21:40:35.421350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.421317 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" event={"ID":"ea9761f2-8a71-4202-881a-13c7572defdd","Type":"ContainerStarted","Data":"ead51785cfbaf34c3419d63055408a00d76f534d9fa05343113841766c036eda"} Oct 30 21:40:35.421665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.421619 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:35.449515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.449479 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:35.460885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.460836 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" podStartSLOduration=1.102892916 podStartE2EDuration="1.460823816s" totalImagesPullingTime="357.9309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:35.460455265 +0000 UTC m=+1874.670394893" watchObservedRunningTime="2026-10-30 21:40:35.460823816 +0000 UTC m=+1874.670763446" Oct 30 21:40:35.945823 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.945787 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz"] Oct 30 21:40:35.951193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.951162 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:35.972431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.971375 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz"] Oct 30 21:40:35.976274 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:35.976242 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e06b90-openshift-default-dockercfg-5tg8v\"" Oct 30 21:40:36.010075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010038 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010106 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010154 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010207 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010237 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010285 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010317 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfk9l\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010448 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.010506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.010493 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.045386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.045352 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:36.111327 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111301 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111446 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111491 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111529 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111573 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111600 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111649 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111679 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfk9l\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111780 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.111920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.111899 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.112137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.112110 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.112280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.112251 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.112477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.112452 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.112578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.112498 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.113765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.113749 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.114197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.114175 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.128840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.128813 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.129024 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.129005 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfk9l\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l\") pod \"e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.195110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.195038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:36.274463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.273644 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:36.506484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.506455 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:36.657466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.657435 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz"] Oct 30 21:40:36.657605 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:36.657522 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b9b2fe_f726_4915_b4a5_6cc2466ad5f7.slice/crio-25ce952da94b416f3665ca7157bb9e3c70e863555ec32624eebb7c6764672fbe WatchSource:0}: Error finding container 25ce952da94b416f3665ca7157bb9e3c70e863555ec32624eebb7c6764672fbe: Status 404 returned error can't find the container with id 25ce952da94b416f3665ca7157bb9e3c70e863555ec32624eebb7c6764672fbe Oct 30 21:40:36.660427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.660371 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:36.660511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.660471 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:36.660553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:36.660516 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:37.445030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:37.444996 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" event={"ID":"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7","Type":"ContainerStarted","Data":"09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611"} Oct 30 21:40:37.445614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:37.445060 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" event={"ID":"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7","Type":"ContainerStarted","Data":"25ce952da94b416f3665ca7157bb9e3c70e863555ec32624eebb7c6764672fbe"} Oct 30 21:40:37.490927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:37.490880 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podStartSLOduration=2.490865856 podStartE2EDuration="2.490865856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:37.490313238 +0000 UTC m=+1876.700252866" watchObservedRunningTime="2026-10-30 21:40:37.490865856 +0000 UTC m=+1876.700805484" Oct 30 21:40:38.267473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:38.267436 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:38.274438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:38.274383 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:38.282786 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:38.282763 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:38.450751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:38.450720 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:38.451485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:38.451467 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:40:39.930982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.930952 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66"] Oct 30 21:40:39.936082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.936059 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:39.959350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.959325 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3172e5-serving-cert\"" Oct 30 21:40:39.963962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.963940 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66"] Oct 30 21:40:39.964892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.964791 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:39.965039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.964908 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:39.965039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.965001 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6h57\" (UniqueName: \"kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:39.965146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:39.965118 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.066494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.066464 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.066494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.066496 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.066672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.066553 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6h57\" (UniqueName: \"kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.066672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.066621 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.066740 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:40:40.066672 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-serving-cert: secret "maas-api-e2e-shared-a-w0-3172e5-serving-cert" not found Oct 30 21:40:40.066740 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:40:40.066734 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls podName:d8473744-30a3-4ea3-8e50-9fd5ad981ff0 nodeName:}" failed. No retries permitted until 2026-10-30 21:40:40.566719019 +0000 UTC m=+1879.776658639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls") pod "maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" (UID: "d8473744-30a3-4ea3-8e50-9fd5ad981ff0") : secret "maas-api-e2e-shared-a-w0-3172e5-serving-cert" not found Oct 30 21:40:40.067170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.067136 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.069037 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.069013 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.088645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.088622 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:40.092046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.092026 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6h57\" (UniqueName: \"kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.247528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.247441 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:40.575479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.575365 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.578050 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.578027 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:40.849067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:40.848975 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:41.189033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.189001 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66"] Oct 30 21:40:41.189610 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:41.189581 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8473744_30a3_4ea3_8e50_9fd5ad981ff0.slice/crio-cd4c9bb8431ce4ea087a98c2c33290f078709c43ec8d51ef251f66a34ba5c7fe WatchSource:0}: Error finding container cd4c9bb8431ce4ea087a98c2c33290f078709c43ec8d51ef251f66a34ba5c7fe: Status 404 returned error can't find the container with id cd4c9bb8431ce4ea087a98c2c33290f078709c43ec8d51ef251f66a34ba5c7fe Oct 30 21:40:41.271775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.251928 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:41.287032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.287003 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:41.441128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.441058 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 21:40:41.470019 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.469990 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" event={"ID":"d8473744-30a3-4ea3-8e50-9fd5ad981ff0","Type":"ContainerStarted","Data":"cd4c9bb8431ce4ea087a98c2c33290f078709c43ec8d51ef251f66a34ba5c7fe"} Oct 30 21:40:41.481517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.481494 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:40:41.865680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:41.865592 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:42.018328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.018283 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:42.116310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.116220 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4"] Oct 30 21:40:42.130410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.122239 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.142203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.142170 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4"] Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204288 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204358 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204426 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204451 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbpl\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204523 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204548 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204674 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204786 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.208481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.204814 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306497 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306533 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbpl\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306586 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306603 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306650 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306732 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306760 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.306919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306861 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.307069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.306919 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.307069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.307028 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.307170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.307073 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.307265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.307244 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.307315 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.307280 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.307539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.307516 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.309280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.309215 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.309467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.309362 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:42.309581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.309564 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.323705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.323682 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.324210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.324168 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbpl\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.444686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.444650 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:42.479558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.479516 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" event={"ID":"d8473744-30a3-4ea3-8e50-9fd5ad981ff0","Type":"ContainerStarted","Data":"f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19"} Oct 30 21:40:42.479688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.479581 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:42.524744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.524555 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" podStartSLOduration=3.231366038 podStartE2EDuration="3.524528592s" totalImagesPullingTime="293.162554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:42.519930406 +0000 UTC m=+1881.729870036" watchObservedRunningTime="2026-10-30 21:40:42.524528592 +0000 UTC m=+1881.734468222" Oct 30 21:40:42.756939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.756911 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4"] Oct 30 21:40:42.758180 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:40:42.758156 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9227f671_9a58_4202_a58a_8338b7c044da.slice/crio-76db3e5dfbbcd85dce9cf62eb98f457280f3a498164db91e2b46bddfe3c809c3 WatchSource:0}: Error finding container 76db3e5dfbbcd85dce9cf62eb98f457280f3a498164db91e2b46bddfe3c809c3: Status 404 returned error can't find the container with id 76db3e5dfbbcd85dce9cf62eb98f457280f3a498164db91e2b46bddfe3c809c3 Oct 30 21:40:42.761010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.760974 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:42.761104 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.761043 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:42.761104 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:42.761070 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:43.499442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:43.498858 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" event={"ID":"9227f671-9a58-4202-a58a-8338b7c044da","Type":"ContainerStarted","Data":"888d531cb72b8b27aa549d5103c07037de5e76167efc0c79090bef09f094fc64"} Oct 30 21:40:43.499442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:43.498921 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" event={"ID":"9227f671-9a58-4202-a58a-8338b7c044da","Type":"ContainerStarted","Data":"76db3e5dfbbcd85dce9cf62eb98f457280f3a498164db91e2b46bddfe3c809c3"} Oct 30 21:40:43.582416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:43.578787 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podStartSLOduration=1.578764301 podStartE2EDuration="1.578764301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:43.548178612 +0000 UTC m=+1882.758118241" watchObservedRunningTime="2026-10-30 21:40:43.578764301 +0000 UTC m=+1882.788703930" Oct 30 21:40:44.085441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:44.084865 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.242032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:44.241995 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.397880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:44.397843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.445753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:44.445723 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:44.447196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:44.447175 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:40:44.447278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:44.447228 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:40:45.445285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:45.445255 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:40:45.445682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:45.445308 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:40:45.726850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:45.726767 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:46.445144 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:46.445112 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:40:46.445318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:46.445160 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:40:47.384167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:47.384127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:47.445209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:47.445155 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:40:47.445453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:47.445228 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:40:48.445745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:48.445714 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:40:48.446147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:48.445766 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:40:48.504295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:48.504267 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 21:40:48.988029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:48.987987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:49.559954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:49.559928 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:49.560446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:49.560424 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:49.561134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:49.561110 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:40:50.705834 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:50.705796 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:51.657322 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:51.657294 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:40:52.515197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:52.515161 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:54.852528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:54.852481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:55.184049 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:40:55.184013 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:40:56.728122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:56.728069 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:57.210892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:57.210859 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:57.652089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:57.652057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:58.148853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:58.148820 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:58.299416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:58.298917 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:59.003253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:40:59.003212 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:00.759170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:00.759121 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:00.906067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:00.906033 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:02.383337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:02.383305 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:02.533520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:02.533484 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:03.276967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:03.276933 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:04.154731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:04.154690 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:04.319059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:04.319026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:05.592415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:05.592350 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:07.429078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:07.429042 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:07.668320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:07.668281 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:08.074595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.074565 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld"] Oct 30 21:41:08.081726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.081704 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.094164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.094139 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-09eabe-openshift-default-dockercfg-dqwhz\"" Oct 30 21:41:08.107439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.107382 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld"] Oct 30 21:41:08.136501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136469 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136554 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136588 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136624 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136658 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136684 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136735 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136843 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pts27\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.136943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.136873 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.151441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.151417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:08.238518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238492 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238537 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238557 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238578 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238597 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238614 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238642 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238723 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pts27\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.238864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238752 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.239020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.238912 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.239118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.239085 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.239118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.239095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.239219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.239154 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.239275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.239251 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.240792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.240771 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.241009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.240993 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.255739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.255717 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.255862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.255842 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pts27\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.396598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.396569 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:08.516044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.516012 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2"] Oct 30 21:41:08.528720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.528697 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.544453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.542681 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f49db2-openshift-default-dockercfg-p89sf\"" Oct 30 21:41:08.547685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.547659 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2"] Oct 30 21:41:08.627033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.627001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:08.645950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.645921 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.645968 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646000 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646137 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646242 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646278 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646346 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646440 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.646562 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.646548 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzcz\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.706435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.706409 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld"] Oct 30 21:41:08.707029 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:41:08.707005 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570a2609_dedf_4228_9fff_3d18d558f715.slice/crio-e84976f4c0c559883c2de7304b6290dd796d3c8a517567829bfa87c8b6ff262b WatchSource:0}: Error finding container e84976f4c0c559883c2de7304b6290dd796d3c8a517567829bfa87c8b6ff262b: Status 404 returned error can't find the container with id e84976f4c0c559883c2de7304b6290dd796d3c8a517567829bfa87c8b6ff262b Oct 30 21:41:08.709625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.709593 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:08.709705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.709655 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:08.709705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.709683 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:08.748064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748041 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748077 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748100 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748221 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748153 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748229 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748259 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748279 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748315 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.748483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748428 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzcz\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.749075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748777 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.749075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748808 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.749075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.748843 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.749075 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.749017 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.749312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.749214 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.750875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.750718 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.750875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.750814 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.765754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.765734 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzcz\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.766115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.766095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token\") pod \"e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:08.803957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.803672 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:08.879410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:08.871381 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:09.323423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.323364 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2"] Oct 30 21:41:09.324477 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:41:09.324452 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac72929_ed78_471f_8d8d_49f5dde68520.slice/crio-c4fffeaba9e38ea3ab6449ae9a66c98a0504d2928fb0e53e616f23c486dba280 WatchSource:0}: Error finding container c4fffeaba9e38ea3ab6449ae9a66c98a0504d2928fb0e53e616f23c486dba280: Status 404 returned error can't find the container with id c4fffeaba9e38ea3ab6449ae9a66c98a0504d2928fb0e53e616f23c486dba280 Oct 30 21:41:09.329361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.329314 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:09.329665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.329626 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:09.338196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.333980 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:09.698083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.698049 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" event={"ID":"7ac72929-ed78-471f-8d8d-49f5dde68520","Type":"ContainerStarted","Data":"3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8"} Oct 30 21:41:09.698569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.698127 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" event={"ID":"7ac72929-ed78-471f-8d8d-49f5dde68520","Type":"ContainerStarted","Data":"c4fffeaba9e38ea3ab6449ae9a66c98a0504d2928fb0e53e616f23c486dba280"} Oct 30 21:41:09.699738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.699714 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" event={"ID":"570a2609-dedf-4228-9fff-3d18d558f715","Type":"ContainerStarted","Data":"2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5"} Oct 30 21:41:09.699860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.699756 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" event={"ID":"570a2609-dedf-4228-9fff-3d18d558f715","Type":"ContainerStarted","Data":"e84976f4c0c559883c2de7304b6290dd796d3c8a517567829bfa87c8b6ff262b"} Oct 30 21:41:09.727566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.727528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:09.738937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.738887 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podStartSLOduration=1.738871564 podStartE2EDuration="1.738871564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.736770176 +0000 UTC m=+1908.946709804" watchObservedRunningTime="2026-10-30 21:41:09.738871564 +0000 UTC m=+1908.948811199" Oct 30 21:41:09.782266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.782199 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podStartSLOduration=1.782179475 podStartE2EDuration="1.782179475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.782111313 +0000 UTC m=+1908.992050964" watchObservedRunningTime="2026-10-30 21:41:09.782179475 +0000 UTC m=+1908.992119106" Oct 30 21:41:09.882410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:09.876503 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:10.397082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:10.397049 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:10.398269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:10.398243 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.398447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:10.398292 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:41:10.886794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:10.886761 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:11.397606 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:11.397574 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.397758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:11.397624 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:41:11.711644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:11.711560 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:11.712421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:11.712378 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:41:11.883167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:11.883132 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:12.328616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:12.328582 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:12.397629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:12.397596 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:41:12.397791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:12.397642 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:41:12.488865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:12.488826 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:13.139440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:13.139378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:13.398017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:13.397925 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:41:13.398506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:13.397999 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:41:13.598884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:13.598820 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:14.111237 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:14.111203 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:14.346806 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:14.346752 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:14.398417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:14.397152 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:41:14.398417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:14.397223 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:41:14.574414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:14.573001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:15.401920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:15.401890 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:15.402356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:15.402256 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:15.402888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:15.402871 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:41:15.965530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:15.965503 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:41:16.921223 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:16.921186 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:18.456680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:18.456645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:18.636607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:18.636574 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:19.155441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:19.155409 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:20.379835 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:20.379802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:20.643466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:20.641621 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:21.215716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:21.215679 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:22.925072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:22.924983 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:24.492563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:24.492524 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:24.492926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:24.492606 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:24.574260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:24.574226 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:25.183729 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:41:25.183691 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:41:26.248982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.245989 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:26.838345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.838315 5039 generic.go:356] "Generic (PLEG): container finished" podID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerID="09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611" exitCode=0 Oct 30 21:41:26.838538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.838415 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" event={"ID":"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7","Type":"ContainerDied","Data":"09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611"} Oct 30 21:41:26.838785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.838770 5039 scope.go:122] "RemoveContainer" containerID="09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611" Oct 30 21:41:26.840128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.840098 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:26.840222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.840162 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:26.840222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:26.840192 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:27.847686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.847648 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" event={"ID":"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7","Type":"ContainerStarted","Data":"1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09"} Oct 30 21:41:27.889251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.889205 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524"] Oct 30 21:41:27.899091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.897047 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524"] Oct 30 21:41:27.899091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.897156 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:27.919275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.915018 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-bfabe1-serving-cert\"" Oct 30 21:41:27.949287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.949256 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:27.949628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.949601 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:27.949763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.949636 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pldzv\" (UniqueName: \"kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:27.950001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.949984 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:27.993628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:27.993578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:28.051552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.051521 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.051706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.051562 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pldzv\" (UniqueName: \"kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.051706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.051599 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.051706 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.051659 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.052194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.052169 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.054080 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.054061 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.054160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.054065 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.101496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.088840 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pldzv\" (UniqueName: \"kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv\") pod \"maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.142926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.142804 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:28.217245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.217211 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:28.281416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.280599 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:41:28.281416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.281001 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:41:28.281416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.281059 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 21:41:28.534928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.534902 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524"] Oct 30 21:41:28.535529 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:41:28.535505 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc714c6f5_861b_4659_b9fa_d4dcef2f58c3.slice/crio-18d1a0eac6a9086083b3d4946603050de96a8807cc1056651323db810b8a5ce0 WatchSource:0}: Error finding container 18d1a0eac6a9086083b3d4946603050de96a8807cc1056651323db810b8a5ce0: Status 404 returned error can't find the container with id 18d1a0eac6a9086083b3d4946603050de96a8807cc1056651323db810b8a5ce0 Oct 30 21:41:28.856240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:28.856155 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" event={"ID":"c714c6f5-861b-4659-b9fa-d4dcef2f58c3","Type":"ContainerStarted","Data":"18d1a0eac6a9086083b3d4946603050de96a8807cc1056651323db810b8a5ce0"} Oct 30 21:41:29.282417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:29.278847 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:41:29.282417 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:29.278909 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 21:41:29.783298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:29.783264 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.895177 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:29.895140 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" event={"ID":"c714c6f5-861b-4659-b9fa-d4dcef2f58c3","Type":"ContainerStarted","Data":"219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0"} Oct 30 21:41:29.895682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:29.895333 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:29.929567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:29.929518 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" podStartSLOduration=2.583963857 podStartE2EDuration="2.929506094s" totalImagesPullingTime="345.542237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:29.928448286 +0000 UTC m=+1929.138387914" watchObservedRunningTime="2026-10-30 21:41:29.929506094 +0000 UTC m=+1929.139445722" Oct 30 21:41:30.277697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:30.277663 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:41:30.277983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:30.277955 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 21:41:30.940889 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:30.940853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:31.277776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.277696 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:41:31.278015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.277988 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 21:41:31.314950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.314912 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6"] Oct 30 21:41:31.319310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.319289 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.331988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.331965 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c19b75-openshift-default-dockercfg-ww26j\"" Oct 30 21:41:31.345887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.345865 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:31.346761 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.346736 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6"] Oct 30 21:41:31.431603 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431574 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.431765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431636 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.431765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431723 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.431765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431753 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2z4\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.431911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431783 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.431911 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431859 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.431981 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.431936 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.432017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.432007 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.432053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.432037 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.532936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.532867 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.532936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.532910 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.532942 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.532965 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2z4\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.532989 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533006 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533031 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533099 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533141 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533328 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533354 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533433 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533586 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.533910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.533891 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.535280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.535257 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.535565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.535546 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.550384 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.550361 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2z4\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.550488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.550442 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.633682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.633657 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:31.693378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.693332 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:31.852972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:31.852759 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:32.100110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.095456 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6"] Oct 30 21:41:32.199938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.199899 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:32.200422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.200401 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:32.200560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.200546 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:32.282500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.280685 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:41:32.282500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.280741 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 21:41:32.337076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.336552 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:32.953749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.953685 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" event={"ID":"7f873ead-7ec5-46bf-b39b-89f5b12039c2","Type":"ContainerStarted","Data":"0cac33da99d929c3531ceb004806b1aad09dd31e6acbeda1f0b27ded1f72a3d9"} Oct 30 21:41:32.953926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:32.953875 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" event={"ID":"7f873ead-7ec5-46bf-b39b-89f5b12039c2","Type":"ContainerStarted","Data":"2fee54df0c74ede61c98cf7790d6b2b474a14fb7ba151a60a300a7419c20b8fb"} Oct 30 21:41:33.005479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.005426 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podStartSLOduration=2.005408257 podStartE2EDuration="2.005408257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:33.004700005 +0000 UTC m=+1932.214639636" watchObservedRunningTime="2026-10-30 21:41:33.005408257 +0000 UTC m=+1932.215347887" Oct 30 21:41:33.279422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.279331 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:41:33.280890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.280856 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:41:33.281491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.281468 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 21:41:33.481413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.480952 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:33.634516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.634440 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:33.636122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.636100 5039 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:41:33.636206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:33.636146 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:41:34.634954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:34.634922 5039 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:41:34.635344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:34.634974 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:41:35.388613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:35.388579 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:35.547257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:35.547220 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:35.634784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:35.634750 5039 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:41:35.635100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:35.635057 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:41:35.913264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:35.913237 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 21:41:36.633978 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:36.633941 5039 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:41:36.634145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:36.633991 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:41:37.095258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:37.095219 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:37.484837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:37.484790 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:37.640158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:37.640119 5039 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:41:37.640361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:37.640181 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:41:37.645948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:37.645918 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:38.634679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:38.634642 5039 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.635555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:38.635515 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:41:39.227642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:39.227593 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:39.638832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:39.638807 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:39.639269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:39.639250 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:39.639891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:39.639872 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:41:40.779968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:40.779934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:42.394219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:42.394186 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:43.809775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.809736 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q"] Oct 30 21:41:43.815364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.815339 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:43.839781 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.839266 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-ae6492-serving-cert\"" Oct 30 21:41:43.844161 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.844134 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q"] Oct 30 21:41:43.905221 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.905193 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9f9\" (UniqueName: \"kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:43.905378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.905364 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:43.905453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.905386 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:43.905496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.905473 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:43.949651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:43.949594 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:44.010249 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.008992 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.010249 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.009041 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.010249 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.009155 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.010249 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.009431 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9f9\" (UniqueName: \"kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.010614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.010292 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.013799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.013776 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.015643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.015609 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.027811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.027778 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9f9\" (UniqueName: \"kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9\") pod \"maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.085727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.085622 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:44.130673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.130643 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:44.426134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:44.426100 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q"] Oct 30 21:41:44.426765 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:41:44.426744 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46047522_2428_48c0_a1d7_adb65bd728e4.slice/crio-51048b21a18c4dc791576986668896ecc507cecc93ed2fc01ef1a2ddfac508ec WatchSource:0}: Error finding container 51048b21a18c4dc791576986668896ecc507cecc93ed2fc01ef1a2ddfac508ec: Status 404 returned error can't find the container with id 51048b21a18c4dc791576986668896ecc507cecc93ed2fc01ef1a2ddfac508ec Oct 30 21:41:45.065727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:45.065691 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" event={"ID":"46047522-2428-48c0-a1d7-adb65bd728e4","Type":"ContainerStarted","Data":"1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24"} Oct 30 21:41:45.066107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:45.065749 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" event={"ID":"46047522-2428-48c0-a1d7-adb65bd728e4","Type":"ContainerStarted","Data":"51048b21a18c4dc791576986668896ecc507cecc93ed2fc01ef1a2ddfac508ec"} Oct 30 21:41:45.066595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:45.066576 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:45.102230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:45.102109 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" podStartSLOduration=1.695862569 podStartE2EDuration="2.102089856s" totalImagesPullingTime="406.227287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:45.09974337 +0000 UTC m=+1944.309682998" watchObservedRunningTime="2026-10-30 21:41:45.102089856 +0000 UTC m=+1944.312029499" Oct 30 21:41:45.701414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:45.701350 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:47.246151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:47.246112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:48.822259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:48.822223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:49.260856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:49.260825 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:41:50.788518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:50.788480 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:52.087492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:52.087463 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 21:41:52.820016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:52.819970 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:53.199030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.198992 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4"] Oct 30 21:41:53.203555 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.203532 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.229351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.229327 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4"] Oct 30 21:41:53.399162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399128 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399325 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399196 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xs45\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399325 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399247 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399325 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399275 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399368 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399427 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399451 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399564 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.399728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.399632 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500236 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500304 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500334 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500456 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500515 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500588 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500612 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xs45\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500644 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500677 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.500908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.500853 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.501089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.501036 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.501215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.501103 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.501340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.501313 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.501687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.501665 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.503728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.503614 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.503819 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.503764 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.518815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.518797 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.519515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.519493 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xs45\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:53.817775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:53.817703 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:54.154086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:54.154061 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4"] Oct 30 21:41:54.155141 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:41:54.155118 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9a100a_efbc_4fe2_a74a_c12ffbfb326d.slice/crio-375dfe126b3ab2292e045a0b4c21dc7a02dce7dbb0916c482b6fb77433c62d38 WatchSource:0}: Error finding container 375dfe126b3ab2292e045a0b4c21dc7a02dce7dbb0916c482b6fb77433c62d38: Status 404 returned error can't find the container with id 375dfe126b3ab2292e045a0b4c21dc7a02dce7dbb0916c482b6fb77433c62d38 Oct 30 21:41:54.181410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:54.157609 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:54.181410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:54.157689 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:54.181410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:54.157733 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:54.712112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:54.712074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:55.136569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:55.136532 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" event={"ID":"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d","Type":"ContainerStarted","Data":"6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a"} Oct 30 21:41:55.136870 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:55.136848 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" event={"ID":"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d","Type":"ContainerStarted","Data":"375dfe126b3ab2292e045a0b4c21dc7a02dce7dbb0916c482b6fb77433c62d38"} Oct 30 21:41:55.218275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:55.218181 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podStartSLOduration=2.21816242 podStartE2EDuration="2.21816242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.200018752 +0000 UTC m=+1954.409958383" watchObservedRunningTime="2026-10-30 21:41:55.21816242 +0000 UTC m=+1954.428102051" Oct 30 21:41:55.235909 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:41:55.235873 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:41:55.818027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:55.817983 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:41:55.820289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:55.820258 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:41:55.820446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:55.820322 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:41:56.720241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:56.720207 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:56.818253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:56.818222 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:41:56.818746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:56.818268 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:41:57.769424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:57.767509 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:57.769424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:57.767614 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:57.819009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:57.818976 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:41:57.819372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:57.819024 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:41:58.302884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:58.302803 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.462415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:58.458127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.818661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:58.818635 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:41:58.818878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:58.818856 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:41:59.778235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:59.778199 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:59.818579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:59.818543 5039 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:41:59.818741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:59.818603 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:41:59.989898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:41:59.989862 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:00.124547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:00.124472 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:00.822675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:00.822615 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:42:00.823137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:00.822969 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:42:00.823724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:00.823706 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:42:00.976656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:00.976618 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:01.207306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.207275 5039 generic.go:356] "Generic (PLEG): container finished" podID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerID="3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8" exitCode=0 Oct 30 21:42:01.207500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.207350 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" event={"ID":"7ac72929-ed78-471f-8d8d-49f5dde68520","Type":"ContainerDied","Data":"3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8"} Oct 30 21:42:01.207910 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.207885 5039 scope.go:122] "RemoveContainer" containerID="3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8" Oct 30 21:42:01.209895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.209857 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:01.210017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.209954 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:01.210017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.210000 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:01.664057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.664016 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:01.822463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.822369 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:01.975110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:01.969995 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:02.218192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:02.218153 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" event={"ID":"7ac72929-ed78-471f-8d8d-49f5dde68520","Type":"ContainerStarted","Data":"52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39"} Oct 30 21:42:02.878416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:02.876685 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:42:02.883048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:02.882542 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.883048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:02.882593 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:42:03.487122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:03.487078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:03.883207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:03.883123 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.883207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:03.883181 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:42:03.896419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:03.896378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:04.037695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:04.037662 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:04.509586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:04.509542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:04.678832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:04.678771 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:04.882511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:04.876724 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.882511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:04.876787 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:42:05.236656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:05.236578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:05.873754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:05.873718 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.874320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:05.874290 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:42:06.878195 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:06.878154 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.878898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:06.878864 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 21:42:07.369894 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:07.369860 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:07.876905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:07.876878 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:42:07.877351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:07.877331 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:42:07.878084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:07.878066 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 21:42:08.298341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:08.298246 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:09.299539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:09.299495 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:10.879918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:10.879884 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.752117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:12.752082 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:14.388638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:14.388602 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:14.524474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:14.524440 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:16.427027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:16.426981 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:18.243088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:18.243047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:19.937849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:19.937814 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:21.704673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:21.704641 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:22.507341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:22.507306 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:23.348854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:23.348818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:24.940844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:24.940804 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:25.185010 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:42:25.184973 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:42:26.128914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.128878 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8"] Oct 30 21:42:26.134437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.134371 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.159472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.159444 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3172e5\"/\"kube-root-ca.crt\"" Oct 30 21:42:26.159632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.159504 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3172e5\"/\"default-dockercfg-lw6j4\"" Oct 30 21:42:26.159725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.159705 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3172e5\"/\"openshift-service-ca.crt\"" Oct 30 21:42:26.159867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.159819 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3172e5\"/\"auth-test-model-3172e5-kserve-self-signed-certs\"" Oct 30 21:42:26.163370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.163350 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8"] Oct 30 21:42:26.238827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.238799 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:26.241215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.241190 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.241340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.241233 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.241423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.241353 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skf26\" (UniqueName: \"kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.241506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.241490 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.241568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.241554 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.241685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.241668 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.342709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.342674 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.342866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.342784 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.342914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.342897 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.342964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.342931 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.343009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.342995 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skf26\" (UniqueName: \"kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.343119 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.343101 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.343375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.343321 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.343511 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.343452 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.343580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.343524 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.345363 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.345337 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.345499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.345379 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.359895 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.359875 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skf26\" (UniqueName: \"kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26\") pod \"auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.451441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.449322 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:26.632643 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.632465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:26.797120 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:42:26.797091 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8109af0_b4d0_4eae_afda_e1e9eaa52721.slice/crio-649e6e894e9183b3a8c6107c3852506aebe1eb048e57a4b8f46f17ac39056b05 WatchSource:0}: Error finding container 649e6e894e9183b3a8c6107c3852506aebe1eb048e57a4b8f46f17ac39056b05: Status 404 returned error can't find the container with id 649e6e894e9183b3a8c6107c3852506aebe1eb048e57a4b8f46f17ac39056b05 Oct 30 21:42:26.797255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:26.797210 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8"] Oct 30 21:42:27.414766 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:27.414729 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" event={"ID":"e8109af0-b4d0-4eae-afda-e1e9eaa52721","Type":"ContainerStarted","Data":"3cfa18234a6b9e251236a37aea0c555bb4ec2deb004cc2b8c2d3170a13f3f07c"} Oct 30 21:42:27.415138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:27.414774 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" event={"ID":"e8109af0-b4d0-4eae-afda-e1e9eaa52721","Type":"ContainerStarted","Data":"649e6e894e9183b3a8c6107c3852506aebe1eb048e57a4b8f46f17ac39056b05"} Oct 30 21:42:27.449857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:27.449828 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:27.463316 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:27.463268 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" podStartSLOduration=1.463254695 podStartE2EDuration="1.463254695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:27.461309363 +0000 UTC m=+1986.671248992" watchObservedRunningTime="2026-10-30 21:42:27.463254695 +0000 UTC m=+1986.673194323" Oct 30 21:42:28.412815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:28.412774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.443354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:30.443322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.400287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:31.400236 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.239313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:32.239276 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:33.970147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:33.970105 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.726381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:34.726337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.889001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:34.888961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:35.031859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:35.031764 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:35.393522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:35.393488 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.449892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:36.449860 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:36.461236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:36.461214 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:36.496815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:36.496792 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:42:36.947210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:36.947172 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:38.589033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:38.588983 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.200757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:40.200693 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.306197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:40.306159 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6"] Oct 30 21:42:40.454864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:40.454787 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:41.790459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:41.790421 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:43.392836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:43.392797 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.484681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:44.484646 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:42:44.870697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:44.870606 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:45.109125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:45.109084 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:46.905461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:46.905426 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.642100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:48.642063 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:50.415832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:50.415791 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.102151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:52.102111 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.749862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:53.749812 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:55.031983 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.031948 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4"] Oct 30 21:42:55.037291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.037269 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.055402 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.055358 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4"] Oct 30 21:42:55.059019 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.058997 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-e06b90\"/\"openshift-service-ca.crt\"" Oct 30 21:42:55.059272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.059248 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-e06b90\"/\"kube-root-ca.crt\"" Oct 30 21:42:55.062014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.061993 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-e06b90\"/\"default-dockercfg-hwvn8\"" Oct 30 21:42:55.062253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.062235 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-e06b90\"/\"auth-test-model-e06b90-kserve-self-signed-certs\"" Oct 30 21:42:55.119966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.119940 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.120174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.120154 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnb5\" (UniqueName: \"kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.120245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.120197 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.120285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.120260 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.120318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.120306 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.120365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.120341 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.130709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.130676 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:55.188268 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:42:55.188236 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:42:55.222015 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.221989 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222069 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222302 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222286 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnb5\" (UniqueName: \"kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222326 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222359 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222450 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222568 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222737 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.222830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.222795 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.224499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.224476 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.224651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.224630 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.239858 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.239834 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnb5\" (UniqueName: \"kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5\") pod \"auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.334891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.334759 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" podUID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" containerName="main" containerID="cri-o://b3b3086572f0f2ac4df701f74994641c20f3b4aec33a07a0b943dbd44bf4ce9b" gracePeriod=45 Oct 30 21:42:55.360408 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.357646 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:55.453020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.452951 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:55.471821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.471559 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" podUID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": dial tcp 10.132.0.112:8000: connect: connection refused" Oct 30 21:42:55.671536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.671506 5039 generic.go:356] "Generic (PLEG): container finished" podID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" containerID="b3b3086572f0f2ac4df701f74994641c20f3b4aec33a07a0b943dbd44bf4ce9b" exitCode=0 Oct 30 21:42:55.671690 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.671556 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" event={"ID":"6969b10c-c154-4cd5-a778-f7d7c6a3549b","Type":"ContainerDied","Data":"b3b3086572f0f2ac4df701f74994641c20f3b4aec33a07a0b943dbd44bf4ce9b"} Oct 30 21:42:55.691681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.691660 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:42:55.737846 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:42:55.737821 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod330af03b_8043_4376_8bf3_02f78c78047d.slice/crio-646cb629a4d0f8f0ce3a7cc0653c5ca9629cdbe16af700a959c819483fccb160 WatchSource:0}: Error finding container 646cb629a4d0f8f0ce3a7cc0653c5ca9629cdbe16af700a959c819483fccb160: Status 404 returned error can't find the container with id 646cb629a4d0f8f0ce3a7cc0653c5ca9629cdbe16af700a959c819483fccb160 Oct 30 21:42:55.745787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.745759 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4"] Oct 30 21:42:55.836362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836342 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home\") pod \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " Oct 30 21:42:55.836486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836383 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h\" (UniqueName: \"kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h\") pod \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " Oct 30 21:42:55.836486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836433 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm\") pod \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " Oct 30 21:42:55.836486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836452 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir\") pod \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " Oct 30 21:42:55.836645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836604 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache\") pod \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " Oct 30 21:42:55.836645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836610 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home" pod "6969b10c-c154-4cd5-a778-f7d7c6a3549b" (UID: "6969b10c-c154-4cd5-a778-f7d7c6a3549b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.836748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836648 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs\") pod \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\" (UID: \"6969b10c-c154-4cd5-a778-f7d7c6a3549b\") " Oct 30 21:42:55.836847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836801 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir" pod "6969b10c-c154-4cd5-a778-f7d7c6a3549b" (UID: "6969b10c-c154-4cd5-a778-f7d7c6a3549b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.836924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.836874 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache" pod "6969b10c-c154-4cd5-a778-f7d7c6a3549b" (UID: "6969b10c-c154-4cd5-a778-f7d7c6a3549b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.837231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.837212 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:42:55.837319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.837256 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:42:55.837319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.837270 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:42:55.840490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.840467 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm" pod "6969b10c-c154-4cd5-a778-f7d7c6a3549b" (UID: "6969b10c-c154-4cd5-a778-f7d7c6a3549b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.840590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.840554 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs" pod "6969b10c-c154-4cd5-a778-f7d7c6a3549b" (UID: "6969b10c-c154-4cd5-a778-f7d7c6a3549b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:55.840674 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.840652 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h" pod "6969b10c-c154-4cd5-a778-f7d7c6a3549b" (UID: "6969b10c-c154-4cd5-a778-f7d7c6a3549b"). InnerVolumeSpecName "kube-api-access-dml8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:55.938331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.938284 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6969b10c-c154-4cd5-a778-f7d7c6a3549b-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:42:55.938331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.938331 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dml8h\" (UniqueName: \"kubernetes.io/projected/6969b10c-c154-4cd5-a778-f7d7c6a3549b-kube-api-access-dml8h\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:42:55.938534 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:55.938346 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6969b10c-c154-4cd5-a778-f7d7c6a3549b-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:42:56.679472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.679433 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" event={"ID":"6969b10c-c154-4cd5-a778-f7d7c6a3549b","Type":"ContainerDied","Data":"5470250f458c972320101e83c407170105011b03555550759d0aaed33d406c70"} Oct 30 21:42:56.679887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.679497 5039 scope.go:122] "RemoveContainer" containerID="b3b3086572f0f2ac4df701f74994641c20f3b4aec33a07a0b943dbd44bf4ce9b" Oct 30 21:42:56.679887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.679454 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6" Oct 30 21:42:56.681443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.681417 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" event={"ID":"330af03b-8043-4376-8bf3-02f78c78047d","Type":"ContainerStarted","Data":"e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9"} Oct 30 21:42:56.681564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.681454 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" event={"ID":"330af03b-8043-4376-8bf3-02f78c78047d","Type":"ContainerStarted","Data":"646cb629a4d0f8f0ce3a7cc0653c5ca9629cdbe16af700a959c819483fccb160"} Oct 30 21:42:56.728313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.728291 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6"] Oct 30 21:42:56.736958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.736937 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-c19b75/praxis-route-c19b75-kserve-64dc9c7895-ftzb6"] Oct 30 21:42:56.765677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.765632 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" podStartSLOduration=1.7656169579999998 podStartE2EDuration="1.765616958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:56.76352392 +0000 UTC m=+2015.973463548" watchObservedRunningTime="2026-10-30 21:42:56.765616958 +0000 UTC m=+2015.975556588" Oct 30 21:42:56.927531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:56.927378 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" path="/var/lib/kubelet/pods/6969b10c-c154-4cd5-a778-f7d7c6a3549b/volumes" Oct 30 21:42:57.263849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:57.263784 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:57.358095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:57.358065 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:42:58.994613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:42:58.994577 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:00.587339 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:00.586231 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:02.177190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:02.177146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:03.826437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:03.824867 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:05.358310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.358276 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:43:05.369542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.369515 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:43:05.525215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.525176 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh"] Oct 30 21:43:05.525967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.525946 5039 cpu_manager.go:414] "containerMap: removing container" podUID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" containerName="main" Oct 30 21:43:05.525967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.525967 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" containerName="main" Oct 30 21:43:05.526177 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.526164 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="6969b10c-c154-4cd5-a778-f7d7c6a3549b" containerName="main" Oct 30 21:43:05.527882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.527854 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:05.531832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.531807 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.563472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563246 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.563472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563338 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.563472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563404 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.563749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563623 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.563749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563650 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96zs\" (UniqueName: \"kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.563857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563750 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.569013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.563981 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f49db2\"/\"auth-test-model-f49db2-kserve-self-signed-certs\"" Oct 30 21:43:05.569013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.564023 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f49db2\"/\"kube-root-ca.crt\"" Oct 30 21:43:05.569013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.564084 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f49db2\"/\"openshift-service-ca.crt\"" Oct 30 21:43:05.569013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.564106 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f49db2\"/\"default-dockercfg-d2xwh\"" Oct 30 21:43:05.571256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.571216 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh"] Oct 30 21:43:05.664896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.664855 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.664936 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.664970 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665088 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.665011 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.665211 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.665239 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96zs\" (UniqueName: \"kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.665491 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.665510 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.665718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.665558 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.667491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.667471 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.667787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.667772 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.691341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.691316 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96zs\" (UniqueName: \"kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs\") pod \"auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.762096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.762068 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:43:05.876798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.874100 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:05.983065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:05.982918 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:06.286210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:06.286181 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh"] Oct 30 21:43:06.286667 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:43:06.286645 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d8562c_8b20_4c70_81f2_7a4433a63ee1.slice/crio-e9d0625fe525e99e8c879b0311412315bb31df8ef185816c4ec8fc716e27a673 WatchSource:0}: Error finding container e9d0625fe525e99e8c879b0311412315bb31df8ef185816c4ec8fc716e27a673: Status 404 returned error can't find the container with id e9d0625fe525e99e8c879b0311412315bb31df8ef185816c4ec8fc716e27a673 Oct 30 21:43:06.759581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:06.759548 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" event={"ID":"d9d8562c-8b20-4c70-81f2-7a4433a63ee1","Type":"ContainerStarted","Data":"bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7"} Oct 30 21:43:06.759989 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:06.759596 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" event={"ID":"d9d8562c-8b20-4c70-81f2-7a4433a63ee1","Type":"ContainerStarted","Data":"e9d0625fe525e99e8c879b0311412315bb31df8ef185816c4ec8fc716e27a673"} Oct 30 21:43:06.807295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:06.807255 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" podStartSLOduration=1.8072417079999998 podStartE2EDuration="1.807241708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:06.803683117 +0000 UTC m=+2026.013622745" watchObservedRunningTime="2026-10-30 21:43:06.807241708 +0000 UTC m=+2026.017181382" Oct 30 21:43:06.880159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:06.880137 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:07.489134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:07.486795 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.218142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:09.218105 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:10.432360 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:10.432328 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:43:11.018596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:11.018564 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:13.117794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:13.117751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.523994 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:14.523961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.258273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:15.258239 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.880413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:15.880370 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:15.891832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:15.891809 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:16.863512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:16.863482 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:43:17.180313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:17.180279 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.516073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:17.514974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:18.976058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:18.976022 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:19.436755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:19.436715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:20.890106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:20.890071 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.534110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:22.534071 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:24.213055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:24.213018 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:24.417093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:24.417035 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:25.185915 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:43:25.185882 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:43:25.594022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:25.593937 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:25.785669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:25.785633 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:25.844443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:25.844348 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6"] Oct 30 21:43:25.844631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:25.844602 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" containerID="cri-o://0cac33da99d929c3531ceb004806b1aad09dd31e6acbeda1f0b27ded1f72a3d9" gracePeriod=30 Oct 30 21:43:26.034413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:26.026689 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:26.177424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:26.177376 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:27.949712 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:27.949676 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:29.772780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:29.772738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.985902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:30.985872 5039 generic.go:356] "Generic (PLEG): container finished" podID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerID="0cac33da99d929c3531ceb004806b1aad09dd31e6acbeda1f0b27ded1f72a3d9" exitCode=0 Oct 30 21:43:30.986246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:30.985944 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" event={"ID":"7f873ead-7ec5-46bf-b39b-89f5b12039c2","Type":"ContainerDied","Data":"0cac33da99d929c3531ceb004806b1aad09dd31e6acbeda1f0b27ded1f72a3d9"} Oct 30 21:43:31.139582 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.139453 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.188991 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189040 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189107 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189131 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189185 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189264 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189294 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189350 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.189428 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4\") pod \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\" (UID: \"7f873ead-7ec5-46bf-b39b-89f5b12039c2\") " Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.190800 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:31.192412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.190977 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:31.200830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.200804 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:31.204483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.204458 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:31.204665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.204620 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:31.204916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.204891 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:31.222922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.222492 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:43:31.222922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.222548 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:31.222922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.222750 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4" pod "7f873ead-7ec5-46bf-b39b-89f5b12039c2" (UID: "7f873ead-7ec5-46bf-b39b-89f5b12039c2"). InnerVolumeSpecName "kube-api-access-fr2z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:31.291929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291891 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.291929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291929 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291943 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291957 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2z4\" (UniqueName: \"kubernetes.io/projected/7f873ead-7ec5-46bf-b39b-89f5b12039c2-kube-api-access-fr2z4\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291972 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291985 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.291998 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.292010 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.292133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.292023 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f873ead-7ec5-46bf-b39b-89f5b12039c2-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:43:31.849568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.849536 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:31.994121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.994092 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" Oct 30 21:43:31.994567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.994121 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6" event={"ID":"7f873ead-7ec5-46bf-b39b-89f5b12039c2","Type":"ContainerDied","Data":"2fee54df0c74ede61c98cf7790d6b2b474a14fb7ba151a60a300a7419c20b8fb"} Oct 30 21:43:31.994567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:31.994184 5039 scope.go:122] "RemoveContainer" containerID="0cac33da99d929c3531ceb004806b1aad09dd31e6acbeda1f0b27ded1f72a3d9" Oct 30 21:43:32.068882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:32.068855 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6"] Oct 30 21:43:32.074830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:32.074808 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-628l6"] Oct 30 21:43:32.922818 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:32.922780 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" path="/var/lib/kubelet/pods/7f873ead-7ec5-46bf-b39b-89f5b12039c2/volumes" Oct 30 21:43:33.657253 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:33.657217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:35.466493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:35.466457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:37.122839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:37.122789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:37.324601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:37.324560 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:37.914806 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:37.914764 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:38.755979 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:38.755948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:38.958269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:38.958220 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.369199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:39.369163 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.569488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:39.569447 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.805215 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:39.805177 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:40.032535 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:40.032503 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t"] Oct 30 21:43:40.041633 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:40.041606 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ef017f-g2s9t"] Oct 30 21:43:40.922285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:40.922250 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc904c88-df88-4902-9d09-0581446bdfb7" path="/var/lib/kubelet/pods/dc904c88-df88-4902-9d09-0581446bdfb7/volumes" Oct 30 21:43:41.839046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:41.838992 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:43.749280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:43.749242 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:45.870243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:45.870202 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:46.991745 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:46.991709 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.190537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:47.190491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.437411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:47.432353 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.676460 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:47.676430 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:49.637403 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:49.637367 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:50.618899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:50.618862 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:51.464175 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:51.464092 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.197721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:53.197672 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:54.958565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:54.958524 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:55.185216 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:43:55.185182 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:43:56.711459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:56.711422 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.052104 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:57.052030 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.660600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:57.660565 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.991369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:57.991291 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:59.118070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:43:59.118009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:00.164770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:00.164733 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:44:01.122347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:01.122311 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:01.373506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:01.365098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:02.942804 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:02.942765 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:03.204645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:03.204479 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:03.666765 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:03.666718 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:05.489169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:05.489130 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:07.139902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:07.139857 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:08.791591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:08.791557 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.379906 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:10.379875 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:11.983329 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:11.983293 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:13.507665 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:13.507490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:14.624354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:14.624319 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:15.136771 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:15.136735 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:16.703638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:16.703599 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:18.309657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:18.309610 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:19.966626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:19.966588 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:21.216897 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:21.216870 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:44:21.227506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:21.227481 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:44:21.513992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:21.513924 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:23.183598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:23.183498 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:24.779489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:24.779452 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:24.931751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:24.931715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.185165 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:44:25.185134 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:44:26.579180 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:26.579142 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:28.300748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:28.300696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:30.055789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:30.055746 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:31.596885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:31.596853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:33.243377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:33.243280 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:34.817439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:34.817384 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.589475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:36.589437 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:38.088203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:38.088169 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:39.613443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:39.613376 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:41.217262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:41.217199 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:42.796845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:42.796803 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:44.485148 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:44.485114 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.861439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:45.856509 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:44:46.409010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:46.408965 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:48.500970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:48.496339 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:50.104001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:50.103961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:51.849581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:51.849541 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:53.494441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:53.494376 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:54.992334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:54.992296 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:55.185135 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:44:55.185104 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:44:56.581618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:56.581579 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:57.016861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:57.016824 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.208849 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:58.208814 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.808330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.808295 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.871493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.871463 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq"] Oct 30 21:44:59.872207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.872183 5039 cpu_manager.go:414] "containerMap: removing container" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" Oct 30 21:44:59.872207 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.872204 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" Oct 30 21:44:59.872448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.872433 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f873ead-7ec5-46bf-b39b-89f5b12039c2" containerName="istio-proxy" Oct 30 21:44:59.876126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.876104 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.897181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.897157 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq"] Oct 30 21:44:59.900067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.900040 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a2fc47\"/\"kube-root-ca.crt\"" Oct 30 21:44:59.900181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.900120 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a2fc47\"/\"default-dockercfg-wwdjh\"" Oct 30 21:44:59.900240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.900189 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a2fc47\"/\"openshift-service-ca.crt\"" Oct 30 21:44:59.900332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.900316 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a2fc47\"/\"explicit-model-a2fc47-kserve-self-signed-certs\"" Oct 30 21:44:59.959294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.959261 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.959443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.959411 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.959500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.959466 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.959626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.959614 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.959663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.959643 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.959718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.959676 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblt6\" (UniqueName: \"kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:44:59.986141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:44:59.986105 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.060891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.060819 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.060891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.060862 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061038 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061083 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061135 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblt6\" (UniqueName: \"kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061345 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061256 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061205 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061623 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.061725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.061653 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.063770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.063747 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.063997 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.063980 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.077824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.077794 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblt6\" (UniqueName: \"kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6\") pod \"explicit-model-a2fc47-kserve-dc8b76885-c9ltq\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.195410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.194681 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:00.248463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.245025 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp"] Oct 30 21:45:00.255969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.254847 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.312611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.312545 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw"] Oct 30 21:45:00.320412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.318934 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54"] Oct 30 21:45:00.322595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.322575 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.324605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.324587 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.332811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.332156 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\"" Oct 30 21:45:00.348257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.348173 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp"] Oct 30 21:45:00.353593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.353545 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw"] Oct 30 21:45:00.360309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.357838 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9"] Oct 30 21:45:00.362710 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.362580 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.364798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.364768 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54"] Oct 30 21:45:00.380491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.380473 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9"] Oct 30 21:45:00.385990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.385504 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhp24\" (UniqueName: \"kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.385990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.385759 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.487332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487298 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.487547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487461 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.487547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487525 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.487663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487627 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhg7d\" (UniqueName: \"kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.487854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487829 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhp24\" (UniqueName: \"kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.487965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487900 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsmc\" (UniqueName: \"kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.487965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.487949 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.488072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.488035 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4hz\" (UniqueName: \"kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.488469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.488440 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.504292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.504264 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhp24\" (UniqueName: \"kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.589660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.589589 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhg7d\" (UniqueName: \"kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.589660 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.589655 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsmc\" (UniqueName: \"kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.589833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.589690 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.589833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.589742 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4hz\" (UniqueName: \"kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.589833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.589802 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.590002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.589856 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.590370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.590349 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.590445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.590431 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.590494 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.590478 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.606963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.606935 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4hz\" (UniqueName: \"kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.607062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.607008 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsmc\" (UniqueName: \"kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.607062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.607022 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhg7d\" (UniqueName: \"kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.612866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.612848 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:00.643520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.643495 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq"] Oct 30 21:45:00.644290 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:45:00.644263 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c52fd18_675a_4b47_a3ff_b7b4199bd5e3.slice/crio-55514c5f2ce66d5c7c8f23d330d2e4144135308599ea3abbb7c5ccac65f7611c WatchSource:0}: Error finding container 55514c5f2ce66d5c7c8f23d330d2e4144135308599ea3abbb7c5ccac65f7611c: Status 404 returned error can't find the container with id 55514c5f2ce66d5c7c8f23d330d2e4144135308599ea3abbb7c5ccac65f7611c Oct 30 21:45:00.672661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.672638 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:00.687604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.685155 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:00.687604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.685662 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" event={"ID":"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3","Type":"ContainerStarted","Data":"55514c5f2ce66d5c7c8f23d330d2e4144135308599ea3abbb7c5ccac65f7611c"} Oct 30 21:45:00.701264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.701239 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:00.917754 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:45:00.917729 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b8025a_5df4_404f_b60c_54ff2b23a014.slice/crio-35e78969dc0a13b62d5b6b126e3098c430bcb2e026c97d91f323279f461b2865 WatchSource:0}: Error finding container 35e78969dc0a13b62d5b6b126e3098c430bcb2e026c97d91f323279f461b2865: Status 404 returned error can't find the container with id 35e78969dc0a13b62d5b6b126e3098c430bcb2e026c97d91f323279f461b2865 Oct 30 21:45:00.930499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.930473 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp"] Oct 30 21:45:00.991109 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:45:00.991084 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc669e4_6495_4c9a_ba02_d83ae7bcaa6e.slice/crio-0853a7e9fc087106431f20b7c8f5c80fac4b0ef5a8ab9522cceec62cebbe5b28 WatchSource:0}: Error finding container 0853a7e9fc087106431f20b7c8f5c80fac4b0ef5a8ab9522cceec62cebbe5b28: Status 404 returned error can't find the container with id 0853a7e9fc087106431f20b7c8f5c80fac4b0ef5a8ab9522cceec62cebbe5b28 Oct 30 21:45:00.991273 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:00.991250 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw"] Oct 30 21:45:01.036006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.035178 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54"] Oct 30 21:45:01.046563 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:45:01.046507 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e2278c_830b_42e1_b81c_aae104bffa53.slice/crio-d7825315d68d07bd54fd35bfae45bb732b3034af69a5c6173667e146da419bb7 WatchSource:0}: Error finding container d7825315d68d07bd54fd35bfae45bb732b3034af69a5c6173667e146da419bb7: Status 404 returned error can't find the container with id d7825315d68d07bd54fd35bfae45bb732b3034af69a5c6173667e146da419bb7 Oct 30 21:45:01.048714 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.048693 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9"] Oct 30 21:45:01.693970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.693939 5039 generic.go:356] "Generic (PLEG): container finished" podID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerID="a15cc656314852cae3d6754edc2d812ae80dce23766130ab306df8b2f4481247" exitCode=60 Oct 30 21:45:01.694079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.694006 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" event={"ID":"31b8025a-5df4-404f-b60c-54ff2b23a014","Type":"ContainerDied","Data":"a15cc656314852cae3d6754edc2d812ae80dce23766130ab306df8b2f4481247"} Oct 30 21:45:01.694079 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.694060 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" event={"ID":"31b8025a-5df4-404f-b60c-54ff2b23a014","Type":"ContainerStarted","Data":"35e78969dc0a13b62d5b6b126e3098c430bcb2e026c97d91f323279f461b2865"} Oct 30 21:45:01.694269 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.694252 5039 scope.go:122] "RemoveContainer" containerID="a15cc656314852cae3d6754edc2d812ae80dce23766130ab306df8b2f4481247" Oct 30 21:45:01.695748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.695716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:01.696118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.696098 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerStarted","Data":"4dea24fe0fcd8f8842feb72b886e5009801bffd0a6193c67b76306933b0b43a5"} Oct 30 21:45:01.696227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.696148 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerStarted","Data":"44e865b2d39eb5b947f74632759b316ce822981bbc2d95d482138d8308ee6d87"} Oct 30 21:45:01.701454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.697958 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerStarted","Data":"98aa2fc7448b53bf82c7ff953b9fea9df0956a6aeff2350919ae3eeb765e41f2"} Oct 30 21:45:01.701454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.698078 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerStarted","Data":"d7825315d68d07bd54fd35bfae45bb732b3034af69a5c6173667e146da419bb7"} Oct 30 21:45:01.701454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.700487 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerStarted","Data":"6518e33a842cf93da99a9e971652fb918b4809e662f1e78bc07056f499341c68"} Oct 30 21:45:01.701454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.700526 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerStarted","Data":"0853a7e9fc087106431f20b7c8f5c80fac4b0ef5a8ab9522cceec62cebbe5b28"} Oct 30 21:45:01.702991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.702968 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" event={"ID":"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3","Type":"ContainerStarted","Data":"d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249"} Oct 30 21:45:01.796450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.796381 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" podStartSLOduration=2.796368399 podStartE2EDuration="2.796368399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.793316124 +0000 UTC m=+2141.003255752" watchObservedRunningTime="2026-10-30 21:45:01.796368399 +0000 UTC m=+2141.006308024" Oct 30 21:45:01.828029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.827972 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" podStartSLOduration=1.827954866 podStartE2EDuration="1.827954866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.826192016 +0000 UTC m=+2141.036131644" watchObservedRunningTime="2026-10-30 21:45:01.827954866 +0000 UTC m=+2141.037894495" Oct 30 21:45:01.861385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.861343 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" podStartSLOduration=1.8613316819999999 podStartE2EDuration="1.861331682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.858183141 +0000 UTC m=+2141.068122770" watchObservedRunningTime="2026-10-30 21:45:01.861331682 +0000 UTC m=+2141.071271310" Oct 30 21:45:01.890385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:01.890336 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" podStartSLOduration=1.890322482 podStartE2EDuration="1.890322482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.889704318 +0000 UTC m=+2141.099643963" watchObservedRunningTime="2026-10-30 21:45:01.890322482 +0000 UTC m=+2141.100262168" Oct 30 21:45:02.196358 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.196335 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:02.711219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.711186 5039 generic.go:356] "Generic (PLEG): container finished" podID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerID="98aa2fc7448b53bf82c7ff953b9fea9df0956a6aeff2350919ae3eeb765e41f2" exitCode=60 Oct 30 21:45:02.711379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.711233 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerDied","Data":"98aa2fc7448b53bf82c7ff953b9fea9df0956a6aeff2350919ae3eeb765e41f2"} Oct 30 21:45:02.711570 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.711554 5039 scope.go:122] "RemoveContainer" containerID="98aa2fc7448b53bf82c7ff953b9fea9df0956a6aeff2350919ae3eeb765e41f2" Oct 30 21:45:02.712982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.712958 5039 generic.go:356] "Generic (PLEG): container finished" podID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerID="6518e33a842cf93da99a9e971652fb918b4809e662f1e78bc07056f499341c68" exitCode=60 Oct 30 21:45:02.713096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.713039 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerDied","Data":"6518e33a842cf93da99a9e971652fb918b4809e662f1e78bc07056f499341c68"} Oct 30 21:45:02.713268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.713247 5039 scope.go:122] "RemoveContainer" containerID="6518e33a842cf93da99a9e971652fb918b4809e662f1e78bc07056f499341c68" Oct 30 21:45:02.717154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.717130 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" event={"ID":"31b8025a-5df4-404f-b60c-54ff2b23a014","Type":"ContainerStarted","Data":"5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88"} Oct 30 21:45:02.718768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.718748 5039 generic.go:356] "Generic (PLEG): container finished" podID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerID="4dea24fe0fcd8f8842feb72b886e5009801bffd0a6193c67b76306933b0b43a5" exitCode=60 Oct 30 21:45:02.718850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.718825 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerDied","Data":"4dea24fe0fcd8f8842feb72b886e5009801bffd0a6193c67b76306933b0b43a5"} Oct 30 21:45:02.719029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.719013 5039 scope.go:122] "RemoveContainer" containerID="4dea24fe0fcd8f8842feb72b886e5009801bffd0a6193c67b76306933b0b43a5" Oct 30 21:45:02.810783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:02.810741 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" podStartSLOduration=2.810728829 podStartE2EDuration="2.810728829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.809217619 +0000 UTC m=+2142.019157247" watchObservedRunningTime="2026-10-30 21:45:02.810728829 +0000 UTC m=+2142.020668457" Oct 30 21:45:03.409117 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.409085 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:03.728453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.728410 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerStarted","Data":"5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1"} Oct 30 21:45:03.730359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.730328 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerStarted","Data":"3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd"} Oct 30 21:45:03.732108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.732084 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerStarted","Data":"e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945"} Oct 30 21:45:03.733840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.733815 5039 generic.go:356] "Generic (PLEG): container finished" podID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerID="5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88" exitCode=60 Oct 30 21:45:03.733953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.733883 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" event={"ID":"31b8025a-5df4-404f-b60c-54ff2b23a014","Type":"ContainerDied","Data":"5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88"} Oct 30 21:45:03.733953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.733948 5039 scope.go:122] "RemoveContainer" containerID="a15cc656314852cae3d6754edc2d812ae80dce23766130ab306df8b2f4481247" Oct 30 21:45:03.734211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:03.734130 5039 scope.go:122] "RemoveContainer" containerID="5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88" Oct 30 21:45:03.734488 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:03.734465 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp_odh-ai-gateway-infra(31b8025a-5df4-404f-b60c-54ff2b23a014)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" Oct 30 21:45:04.742324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.742295 5039 scope.go:122] "RemoveContainer" containerID="5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88" Oct 30 21:45:04.742697 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:04.742579 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp_odh-ai-gateway-infra(31b8025a-5df4-404f-b60c-54ff2b23a014)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" Oct 30 21:45:04.744116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.744091 5039 generic.go:356] "Generic (PLEG): container finished" podID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerID="5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1" exitCode=60 Oct 30 21:45:04.744247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.744158 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerDied","Data":"5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1"} Oct 30 21:45:04.744306 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.744245 5039 scope.go:122] "RemoveContainer" containerID="4dea24fe0fcd8f8842feb72b886e5009801bffd0a6193c67b76306933b0b43a5" Oct 30 21:45:04.744372 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.744336 5039 scope.go:122] "RemoveContainer" containerID="5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1" Oct 30 21:45:04.744628 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:04.744605 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54_odh-ai-gateway-infra(cbd05d0e-0b80-4802-81b7-38929ba42ef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" Oct 30 21:45:04.746066 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.746044 5039 generic.go:356] "Generic (PLEG): container finished" podID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerID="3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd" exitCode=60 Oct 30 21:45:04.746169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.746108 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerDied","Data":"3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd"} Oct 30 21:45:04.746346 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.746332 5039 scope.go:122] "RemoveContainer" containerID="3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd" Oct 30 21:45:04.746614 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:04.746595 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9_odh-ai-gateway-infra(d0e2278c-830b-42e1-b81c-aae104bffa53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" Oct 30 21:45:04.747992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.747971 5039 generic.go:356] "Generic (PLEG): container finished" podID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerID="e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945" exitCode=60 Oct 30 21:45:04.748096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.748049 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerDied","Data":"e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945"} Oct 30 21:45:04.748258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.748242 5039 scope.go:122] "RemoveContainer" containerID="e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945" Oct 30 21:45:04.748506 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:04.748485 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw_odh-ai-gateway-infra(cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" Oct 30 21:45:04.756268 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.756252 5039 scope.go:122] "RemoveContainer" containerID="98aa2fc7448b53bf82c7ff953b9fea9df0956a6aeff2350919ae3eeb765e41f2" Oct 30 21:45:04.769648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:04.769621 5039 scope.go:122] "RemoveContainer" containerID="6518e33a842cf93da99a9e971652fb918b4809e662f1e78bc07056f499341c68" Oct 30 21:45:05.037595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:05.037522 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:05.756333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:05.756310 5039 scope.go:122] "RemoveContainer" containerID="5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1" Oct 30 21:45:05.756810 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:05.756586 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54_odh-ai-gateway-infra(cbd05d0e-0b80-4802-81b7-38929ba42ef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" Oct 30 21:45:05.758251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:05.758228 5039 scope.go:122] "RemoveContainer" containerID="3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd" Oct 30 21:45:05.758538 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:05.758512 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9_odh-ai-gateway-infra(d0e2278c-830b-42e1-b81c-aae104bffa53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" Oct 30 21:45:05.759950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:05.759927 5039 scope.go:122] "RemoveContainer" containerID="e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945" Oct 30 21:45:05.760149 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:05.760130 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw_odh-ai-gateway-infra(cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" Oct 30 21:45:06.853425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:06.851062 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq"] Oct 30 21:45:06.886886 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:06.886856 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.069199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:07.069166 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:08.883701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:08.883664 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:10.616589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:10.616543 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:12.129843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:12.129789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:12.914521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:12.914483 5039 scope.go:122] "RemoveContainer" containerID="5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88" Oct 30 21:45:13.817875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:13.817846 5039 generic.go:356] "Generic (PLEG): container finished" podID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerID="a65a27a43c5d8bfed74af6b3d1a47488fd451b2456aee3d4b37a179c247a1b45" exitCode=60 Oct 30 21:45:13.818287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:13.817910 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" event={"ID":"31b8025a-5df4-404f-b60c-54ff2b23a014","Type":"ContainerDied","Data":"a65a27a43c5d8bfed74af6b3d1a47488fd451b2456aee3d4b37a179c247a1b45"} Oct 30 21:45:13.818287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:13.817969 5039 scope.go:122] "RemoveContainer" containerID="5129b91fc999cdd57ecb9cdac7e0f81b22ec345758d2aa807fdb276bec4e1d88" Oct 30 21:45:13.818376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:13.818296 5039 scope.go:122] "RemoveContainer" containerID="a65a27a43c5d8bfed74af6b3d1a47488fd451b2456aee3d4b37a179c247a1b45" Oct 30 21:45:13.818620 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:13.818599 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp_odh-ai-gateway-infra(31b8025a-5df4-404f-b60c-54ff2b23a014)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" Oct 30 21:45:13.963685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:13.963596 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.732872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:14.732841 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.885028 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:14.884996 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp"] Oct 30 21:45:14.914947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:14.914916 5039 scope.go:122] "RemoveContainer" containerID="5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1" Oct 30 21:45:15.054292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.054264 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:15.220587 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.220564 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24\" (UniqueName: \"kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24\") pod \"31b8025a-5df4-404f-b60c-54ff2b23a014\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " Oct 30 21:45:15.220747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.220724 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca\") pod \"31b8025a-5df4-404f-b60c-54ff2b23a014\" (UID: \"31b8025a-5df4-404f-b60c-54ff2b23a014\") " Oct 30 21:45:15.221081 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.221057 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca" pod "31b8025a-5df4-404f-b60c-54ff2b23a014" (UID: "31b8025a-5df4-404f-b60c-54ff2b23a014"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:15.221216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.221203 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b8025a-5df4-404f-b60c-54ff2b23a014-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:15.222380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.222360 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24" pod "31b8025a-5df4-404f-b60c-54ff2b23a014" (UID: "31b8025a-5df4-404f-b60c-54ff2b23a014"). InnerVolumeSpecName "kube-api-access-vhp24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:15.322600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.322546 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhp24\" (UniqueName: \"kubernetes.io/projected/31b8025a-5df4-404f-b60c-54ff2b23a014-kube-api-access-vhp24\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:15.430233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.429238 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:15.834642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.834611 5039 generic.go:356] "Generic (PLEG): container finished" podID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerID="6c6fdac62b25446db125a6b2f78c9591b992e45c5db90e8836d1671ba8e5b80c" exitCode=60 Oct 30 21:45:15.834757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.834690 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerDied","Data":"6c6fdac62b25446db125a6b2f78c9591b992e45c5db90e8836d1671ba8e5b80c"} Oct 30 21:45:15.834757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.834754 5039 scope.go:122] "RemoveContainer" containerID="5aaf6cfc99419d884507a0f4b8509e49e926d85d7d523422952fa20b04af6bb1" Oct 30 21:45:15.835153 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.835134 5039 scope.go:122] "RemoveContainer" containerID="6c6fdac62b25446db125a6b2f78c9591b992e45c5db90e8836d1671ba8e5b80c" Oct 30 21:45:15.835447 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:15.835426 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54_odh-ai-gateway-infra(cbd05d0e-0b80-4802-81b7-38929ba42ef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" Oct 30 21:45:15.836514 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.836491 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" Oct 30 21:45:15.836611 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.836516 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp" event={"ID":"31b8025a-5df4-404f-b60c-54ff2b23a014","Type":"ContainerDied","Data":"35e78969dc0a13b62d5b6b126e3098c430bcb2e026c97d91f323279f461b2865"} Oct 30 21:45:15.847525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.847509 5039 scope.go:122] "RemoveContainer" containerID="a65a27a43c5d8bfed74af6b3d1a47488fd451b2456aee3d4b37a179c247a1b45" Oct 30 21:45:15.904990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.904969 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp"] Oct 30 21:45:15.913793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:15.913772 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889945-lj5qp"] Oct 30 21:45:16.880689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:16.880653 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54"] Oct 30 21:45:16.923656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:16.923621 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" path="/var/lib/kubelet/pods/31b8025a-5df4-404f-b60c-54ff2b23a014/volumes" Oct 30 21:45:17.039280 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.039260 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:17.144669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.144638 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz\" (UniqueName: \"kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz\") pod \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " Oct 30 21:45:17.144839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.144749 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca\") pod \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\" (UID: \"cbd05d0e-0b80-4802-81b7-38929ba42ef4\") " Oct 30 21:45:17.145157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.145119 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca" pod "cbd05d0e-0b80-4802-81b7-38929ba42ef4" (UID: "cbd05d0e-0b80-4802-81b7-38929ba42ef4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.146929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.145525 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd05d0e-0b80-4802-81b7-38929ba42ef4-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:17.148342 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.148305 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz" pod "cbd05d0e-0b80-4802-81b7-38929ba42ef4" (UID: "cbd05d0e-0b80-4802-81b7-38929ba42ef4"). InnerVolumeSpecName "kube-api-access-8t4hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.216236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.216203 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:17.247109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.247066 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4hz\" (UniqueName: \"kubernetes.io/projected/cbd05d0e-0b80-4802-81b7-38929ba42ef4-kube-api-access-8t4hz\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:17.858558 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.858523 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" event={"ID":"cbd05d0e-0b80-4802-81b7-38929ba42ef4","Type":"ContainerDied","Data":"44e865b2d39eb5b947f74632759b316ce822981bbc2d95d482138d8308ee6d87"} Oct 30 21:45:17.858740 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.858590 5039 scope.go:122] "RemoveContainer" containerID="6c6fdac62b25446db125a6b2f78c9591b992e45c5db90e8836d1671ba8e5b80c" Oct 30 21:45:17.858740 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.858533 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54" Oct 30 21:45:17.907909 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.907880 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54"] Oct 30 21:45:17.911750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.911728 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889945-r6s54"] Oct 30 21:45:17.914568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.914550 5039 scope.go:122] "RemoveContainer" containerID="e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945" Oct 30 21:45:17.914854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:17.914836 5039 scope.go:122] "RemoveContainer" containerID="3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd" Oct 30 21:45:18.034842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.034801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.185282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.185253 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.868030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.868002 5039 generic.go:356] "Generic (PLEG): container finished" podID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerID="56e74cbfa6f3f09f3142a8d1bcf091cbdc2c27291fcf90de81391de9fee2cdcd" exitCode=60 Oct 30 21:45:18.868151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.868077 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerDied","Data":"56e74cbfa6f3f09f3142a8d1bcf091cbdc2c27291fcf90de81391de9fee2cdcd"} Oct 30 21:45:18.868211 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.868181 5039 scope.go:122] "RemoveContainer" containerID="3238712f8d13b4792abcd87b84e2d57fcf46f72a6c65367c2c78e2928d430ffd" Oct 30 21:45:18.868655 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.868631 5039 scope.go:122] "RemoveContainer" containerID="56e74cbfa6f3f09f3142a8d1bcf091cbdc2c27291fcf90de81391de9fee2cdcd" Oct 30 21:45:18.869064 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:18.869027 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9_odh-ai-gateway-infra(d0e2278c-830b-42e1-b81c-aae104bffa53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" Oct 30 21:45:18.870443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.870423 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerStarted","Data":"013a02284cf9d529ec7b5e4eddab6bbfb7be5080e565ed076da56bb68712fd81"} Oct 30 21:45:18.921815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:18.921793 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" path="/var/lib/kubelet/pods/cbd05d0e-0b80-4802-81b7-38929ba42ef4/volumes" Oct 30 21:45:19.104699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.104174 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:19.882249 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.882219 5039 generic.go:356] "Generic (PLEG): container finished" podID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerID="013a02284cf9d529ec7b5e4eddab6bbfb7be5080e565ed076da56bb68712fd81" exitCode=60 Oct 30 21:45:19.882438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.882274 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerDied","Data":"013a02284cf9d529ec7b5e4eddab6bbfb7be5080e565ed076da56bb68712fd81"} Oct 30 21:45:19.882438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.882318 5039 scope.go:122] "RemoveContainer" containerID="e0da12c8e2063b6faddf7a91b28016e99ac2bb05bc0bd296c50bcdd0b993b945" Oct 30 21:45:19.882727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.882702 5039 scope.go:122] "RemoveContainer" containerID="013a02284cf9d529ec7b5e4eddab6bbfb7be5080e565ed076da56bb68712fd81" Oct 30 21:45:19.883025 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:19.883000 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw_odh-ai-gateway-infra(cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" Oct 30 21:45:19.913492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.913468 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9"] Oct 30 21:45:19.950361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:19.950331 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw"] Oct 30 21:45:20.070523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.070503 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:20.084787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.084767 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca\") pod \"d0e2278c-830b-42e1-b81c-aae104bffa53\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " Oct 30 21:45:20.084899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.084885 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d\" (UniqueName: \"kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d\") pod \"d0e2278c-830b-42e1-b81c-aae104bffa53\" (UID: \"d0e2278c-830b-42e1-b81c-aae104bffa53\") " Oct 30 21:45:20.085096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.085077 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca" pod "d0e2278c-830b-42e1-b81c-aae104bffa53" (UID: "d0e2278c-830b-42e1-b81c-aae104bffa53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.085479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.085461 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e2278c-830b-42e1-b81c-aae104bffa53-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.086831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.086814 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d" pod "d0e2278c-830b-42e1-b81c-aae104bffa53" (UID: "d0e2278c-830b-42e1-b81c-aae104bffa53"). InnerVolumeSpecName "kube-api-access-qhg7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.186448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.186419 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhg7d\" (UniqueName: \"kubernetes.io/projected/d0e2278c-830b-42e1-b81c-aae104bffa53-kube-api-access-qhg7d\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.867409 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.867346 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:20.900808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.898459 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" event={"ID":"d0e2278c-830b-42e1-b81c-aae104bffa53","Type":"ContainerDied","Data":"d7825315d68d07bd54fd35bfae45bb732b3034af69a5c6173667e146da419bb7"} Oct 30 21:45:20.900808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.898541 5039 scope.go:122] "RemoveContainer" containerID="56e74cbfa6f3f09f3142a8d1bcf091cbdc2c27291fcf90de81391de9fee2cdcd" Oct 30 21:45:20.900808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.898662 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9" Oct 30 21:45:20.959329 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.958821 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9"] Oct 30 21:45:20.964644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:20.964295 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889945-jbxv9"] Oct 30 21:45:20.989370 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:20.989095 5039 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e2278c_830b_42e1_b81c_aae104bffa53.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:21.091561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.091539 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:21.126197 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.126122 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc\" (UniqueName: \"kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc\") pod \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " Oct 30 21:45:21.126340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.126210 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca\") pod \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\" (UID: \"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e\") " Oct 30 21:45:21.127130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.127099 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca" pod "cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" (UID: "cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.128585 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.128562 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc" pod "cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" (UID: "cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e"). InnerVolumeSpecName "kube-api-access-wtsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.228264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.228232 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsmc\" (UniqueName: \"kubernetes.io/projected/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-kube-api-access-wtsmc\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:21.228264 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.228259 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:21.440773 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.440747 5039 scope.go:122] "RemoveContainer" containerID="1b0a7d6f77b8869dac08ba169185f851e3edfdfb138dbb650709b5ba6f3ca523" Oct 30 21:45:21.882926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.882844 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" podUID="1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" containerName="main" containerID="cri-o://d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249" gracePeriod=45 Oct 30 21:45:21.912900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.912881 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" Oct 30 21:45:21.913027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.912895 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw" event={"ID":"cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e","Type":"ContainerDied","Data":"0853a7e9fc087106431f20b7c8f5c80fac4b0ef5a8ab9522cceec62cebbe5b28"} Oct 30 21:45:21.913027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.912975 5039 scope.go:122] "RemoveContainer" containerID="013a02284cf9d529ec7b5e4eddab6bbfb7be5080e565ed076da56bb68712fd81" Oct 30 21:45:21.981927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.981905 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw"] Oct 30 21:45:21.985092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:21.985072 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889945-k9rmw"] Oct 30 21:45:22.172219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.172197 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:22.241685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241653 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6\" (UniqueName: \"kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6\") pod \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " Oct 30 21:45:22.242096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241698 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home\") pod \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " Oct 30 21:45:22.242096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241772 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache\") pod \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " Oct 30 21:45:22.242096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241910 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir\") pod \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " Oct 30 21:45:22.242096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241943 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs\") pod \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " Oct 30 21:45:22.242096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241983 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm\") pod \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\" (UID: \"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3\") " Oct 30 21:45:22.242096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.241998 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home" pod "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" (UID: "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:22.242459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.242137 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir" pod "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" (UID: "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:22.242459 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.242252 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache" pod "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" (UID: "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:22.242770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.242747 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.242873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.242777 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.242873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.242793 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.244190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.244167 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm" pod "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" (UID: "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:22.244476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.244454 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs" pod "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" (UID: "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:22.244566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.244493 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6" pod "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" (UID: "1c52fd18-675a-4b47-a3ff-b7b4199bd5e3"). InnerVolumeSpecName "kube-api-access-dblt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.344203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.344178 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.344203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.344203 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.344350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.344211 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dblt6\" (UniqueName: \"kubernetes.io/projected/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3-kube-api-access-dblt6\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.450485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.450374 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:22.923348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.923310 5039 generic.go:356] "Generic (PLEG): container finished" podID="1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" containerID="d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249" exitCode=0 Oct 30 21:45:22.923348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.923325 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" path="/var/lib/kubelet/pods/cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e/volumes" Oct 30 21:45:22.923607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.923445 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" Oct 30 21:45:22.923857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.923821 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" path="/var/lib/kubelet/pods/d0e2278c-830b-42e1-b81c-aae104bffa53/volumes" Oct 30 21:45:22.924318 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.924300 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" event={"ID":"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3","Type":"ContainerDied","Data":"d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249"} Oct 30 21:45:22.924367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.924330 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq" event={"ID":"1c52fd18-675a-4b47-a3ff-b7b4199bd5e3","Type":"ContainerDied","Data":"55514c5f2ce66d5c7c8f23d330d2e4144135308599ea3abbb7c5ccac65f7611c"} Oct 30 21:45:22.924367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.924350 5039 scope.go:122] "RemoveContainer" containerID="d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249" Oct 30 21:45:22.934759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.934738 5039 scope.go:122] "RemoveContainer" containerID="d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249" Oct 30 21:45:22.935059 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:22.935036 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249\": container with ID starting with d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249 not found: ID does not exist" containerID="d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249" Oct 30 21:45:22.935146 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.935064 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249"} err="failed to get container status \"d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249\": rpc error: code = NotFound desc = could not find container \"d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249\": container with ID starting with d02703eca25eff999fb3d9147614e98a968367fa0c8a236a86bfe83290741249 not found: ID does not exist" Oct 30 21:45:22.967027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.967003 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq"] Oct 30 21:45:22.977162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:22.977140 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a2fc47/explicit-model-a2fc47-kserve-dc8b76885-c9ltq"] Oct 30 21:45:23.986276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:23.986243 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:24.922525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:24.922492 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" path="/var/lib/kubelet/pods/1c52fd18-675a-4b47-a3ff-b7b4199bd5e3/volumes" Oct 30 21:45:25.184044 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:25.183970 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:45:25.619626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:25.619528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:27.195657 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:27.195619 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:28.953852 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:28.953796 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:30.614285 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:30.614245 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.213934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:32.213878 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:33.592559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:33.592524 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:45:33.829014 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:33.828974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:35.597082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:35.597012 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:37.271004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:37.270965 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:38.940279 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:38.940239 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:40.326784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:40.326747 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:41.059260 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:41.059226 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:41.201613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:41.201583 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:41.694373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:41.694342 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:42.983732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:42.983698 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:44.503189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:44.503160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:46.019407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:46.019355 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:47.607883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:47.607842 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:47.774560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:47.774524 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:48.045795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:48.045756 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:48.708255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:48.708215 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:49.194006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:49.193966 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:49.509666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:49.509566 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:49.663219 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:49.663176 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:51.151120 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:51.151083 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:51.336149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:51.336116 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:51.519959 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:51.519863 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:52.513982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:52.513927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:53.097977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:53.097944 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:54.760799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:54.759574 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:55.184181 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:45:55.184147 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:45:56.331928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:56.331891 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:57.958498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:57.958451 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:58.063276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:58.063247 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4"] Oct 30 21:45:58.063483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:58.063459 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" containerID="cri-o://888d531cb72b8b27aa549d5103c07037de5e76167efc0c79090bef09f094fc64" gracePeriod=30 Oct 30 21:45:58.066711 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:58.066679 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x"] Oct 30 21:45:58.066905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:58.066883 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" containerID="cri-o://4b5052c7d949844af315471b3e15372467ae68c6d3c91d99e1bfdd7fe2ef695e" gracePeriod=30 Oct 30 21:45:58.069869 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:58.069843 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr"] Oct 30 21:45:58.070041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:58.070018 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" containerID="cri-o://644f03a57234dd825bf9e1414615affd2c446c5e2eb336e0d693b2705faa92bf" gracePeriod=30 Oct 30 21:45:59.570619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:45:59.570583 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:01.193495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:01.193458 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:02.131343 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:02.131305 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:02.131520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:02.131366 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:02.738522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:02.738439 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:03.250201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.250165 5039 generic.go:356] "Generic (PLEG): container finished" podID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerID="644f03a57234dd825bf9e1414615affd2c446c5e2eb336e0d693b2705faa92bf" exitCode=0 Oct 30 21:46:03.250381 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.250285 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" event={"ID":"9f69bd36-dade-487d-bba3-db4ce7669a44","Type":"ContainerDied","Data":"644f03a57234dd825bf9e1414615affd2c446c5e2eb336e0d693b2705faa92bf"} Oct 30 21:46:03.252187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.252167 5039 generic.go:356] "Generic (PLEG): container finished" podID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerID="4b5052c7d949844af315471b3e15372467ae68c6d3c91d99e1bfdd7fe2ef695e" exitCode=0 Oct 30 21:46:03.252336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.252321 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" event={"ID":"c8dbc5be-53dd-430e-ab1b-bea41e346093","Type":"ContainerDied","Data":"4b5052c7d949844af315471b3e15372467ae68c6d3c91d99e1bfdd7fe2ef695e"} Oct 30 21:46:03.253900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.253879 5039 generic.go:356] "Generic (PLEG): container finished" podID="9227f671-9a58-4202-a58a-8338b7c044da" containerID="888d531cb72b8b27aa549d5103c07037de5e76167efc0c79090bef09f094fc64" exitCode=0 Oct 30 21:46:03.253991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.253964 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" event={"ID":"9227f671-9a58-4202-a58a-8338b7c044da","Type":"ContainerDied","Data":"888d531cb72b8b27aa549d5103c07037de5e76167efc0c79090bef09f094fc64"} Oct 30 21:46:03.380936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.380906 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:46:03.431659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.431636 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:46:03.435735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.435715 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:46:03.489142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489108 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489291 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489264 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489363 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489409 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.489465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489424 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489583 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489539 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489634 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489594 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489690 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489760 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.489833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489809 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:03.489887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489823 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.489942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.489892 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo\") pod \"9f69bd36-dade-487d-bba3-db4ce7669a44\" (UID: \"9f69bd36-dade-487d-bba3-db4ce7669a44\") " Oct 30 21:46:03.490036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.490017 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.490795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.490767 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69bd36-dade-487d-bba3-db4ce7669a44-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.490925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.490799 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.490925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.490812 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.490925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.490824 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.491043 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.490922 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.492029 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.492001 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.492469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.492444 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "kube-api-access-97swq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.492899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.492873 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.494354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.494325 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo" pod "9f69bd36-dade-487d-bba3-db4ce7669a44" (UID: "9f69bd36-dade-487d-bba3-db4ce7669a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:03.591763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.591691 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.591763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.591732 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.591936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.591781 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.591936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.591909 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.591989 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592004 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592040 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592076 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592084 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:03.592189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592102 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592189 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592135 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592221 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592232 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592205 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592265 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592343 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592334 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592455 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592468 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592488 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592509 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592559 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592632 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token\") pod \"c8dbc5be-53dd-430e-ab1b-bea41e346093\" (UID: \"c8dbc5be-53dd-430e-ab1b-bea41e346093\") " Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592681 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592694 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592740 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592749 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo\") pod \"9227f671-9a58-4202-a58a-8338b7c044da\" (UID: \"9227f671-9a58-4202-a58a-8338b7c044da\") " Oct 30 21:46:03.592892 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.592770 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:03.593537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593513 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593542 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593557 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593576 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593589 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593603 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593617 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9227f671-9a58-4202-a58a-8338b7c044da-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593630 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97swq\" (UniqueName: \"kubernetes.io/projected/9f69bd36-dade-487d-bba3-db4ce7669a44-kube-api-access-97swq\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593643 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593656 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593668 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593681 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593698 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69bd36-dade-487d-bba3-db4ce7669a44-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593710 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.593942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.593723 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8dbc5be-53dd-430e-ab1b-bea41e346093-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.594716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.594690 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.594716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.594815 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.594716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.594838 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "kube-api-access-8sjxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.595076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.594952 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "kube-api-access-vrbpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.595076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.594998 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:03.595194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.595107 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.595602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.595586 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo" pod "9227f671-9a58-4202-a58a-8338b7c044da" (UID: "9227f671-9a58-4202-a58a-8338b7c044da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:03.595938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.595917 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo" pod "c8dbc5be-53dd-430e-ab1b-bea41e346093" (UID: "c8dbc5be-53dd-430e-ab1b-bea41e346093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:03.695728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695697 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9227f671-9a58-4202-a58a-8338b7c044da-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695724 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695737 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbpl\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-kube-api-access-vrbpl\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695751 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9227f671-9a58-4202-a58a-8338b7c044da-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695773 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjxq\" (UniqueName: \"kubernetes.io/projected/c8dbc5be-53dd-430e-ab1b-bea41e346093-kube-api-access-8sjxq\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695785 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9227f671-9a58-4202-a58a-8338b7c044da-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695799 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:03.695947 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:03.695816 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8dbc5be-53dd-430e-ab1b-bea41e346093-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:04.262056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.262028 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" Oct 30 21:46:04.262056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.262038 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x" event={"ID":"c8dbc5be-53dd-430e-ab1b-bea41e346093","Type":"ContainerDied","Data":"fa60aa986a19bdb5dc5d511770c01a597cba552eddec8cf34061bbd3e4e657e4"} Oct 30 21:46:04.262520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.262113 5039 scope.go:122] "RemoveContainer" containerID="4b5052c7d949844af315471b3e15372467ae68c6d3c91d99e1bfdd7fe2ef695e" Oct 30 21:46:04.263782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.263740 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" Oct 30 21:46:04.263961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.263825 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4" event={"ID":"9227f671-9a58-4202-a58a-8338b7c044da","Type":"ContainerDied","Data":"76db3e5dfbbcd85dce9cf62eb98f457280f3a498164db91e2b46bddfe3c809c3"} Oct 30 21:46:04.270833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.270806 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" event={"ID":"9f69bd36-dade-487d-bba3-db4ce7669a44","Type":"ContainerDied","Data":"f062a329af53ebfc32e950e6a78f871b056ffa0ef34e4af2fb5a8cc7f535c5c3"} Oct 30 21:46:04.270943 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.270848 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr" Oct 30 21:46:04.273357 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.273341 5039 scope.go:122] "RemoveContainer" containerID="888d531cb72b8b27aa549d5103c07037de5e76167efc0c79090bef09f094fc64" Oct 30 21:46:04.282440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.282424 5039 scope.go:122] "RemoveContainer" containerID="644f03a57234dd825bf9e1414615affd2c446c5e2eb336e0d693b2705faa92bf" Oct 30 21:46:04.318813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.318792 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x"] Oct 30 21:46:04.329209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.329188 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-mhc5x"] Oct 30 21:46:04.363542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.363169 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:04.370853 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.370686 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr"] Oct 30 21:46:04.375138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.375112 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-tdtkr"] Oct 30 21:46:04.411160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.411116 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4"] Oct 30 21:46:04.411362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.411183 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-8fth4"] Oct 30 21:46:04.922794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.922757 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9227f671-9a58-4202-a58a-8338b7c044da" path="/var/lib/kubelet/pods/9227f671-9a58-4202-a58a-8338b7c044da/volumes" Oct 30 21:46:04.923314 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.923293 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" path="/var/lib/kubelet/pods/9f69bd36-dade-487d-bba3-db4ce7669a44/volumes" Oct 30 21:46:04.923797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:04.923778 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" path="/var/lib/kubelet/pods/c8dbc5be-53dd-430e-ab1b-bea41e346093/volumes" Oct 30 21:46:05.914579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:05.914548 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:07.507450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:07.507414 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:07.685195 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:07.685148 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:07.838586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:07.838493 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:08.176443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:08.176381 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:08.411301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:08.411273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:09.198762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:09.198719 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.770544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:10.770512 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:12.391452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:12.391360 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.355941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:13.355904 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.904566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:13.904498 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:14.075738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:14.075705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:14.852500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:14.852460 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.419732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:15.419695 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:16.991752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:16.991715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:17.235041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:17.235009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:18.719182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:18.719121 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:19.318471 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:19.318436 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:20.187584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:20.187549 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:20.774065 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:20.774027 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:21.786680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:21.786472 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.386430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:23.386343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.879833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:23.879743 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:46:25.226685 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:46:25.226644 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:46:25.241621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:25.241586 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.995003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:26.994947 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.785616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:28.785565 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:29.191990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:29.191957 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw"] Oct 30 21:46:29.192178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:29.192142 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" containerID="cri-o://5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d" gracePeriod=30 Oct 30 21:46:30.285242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:30.285203 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:31.946250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:31.946212 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.630244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:33.630187 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:34.074624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.074589 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:34.220878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.220853 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l"] Oct 30 21:46:34.221055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.221034 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" containerID="cri-o://7c0b415c0271672a13ac9b33cea2afa91a2673182ccaeeb170890f2db6125e05" gracePeriod=30 Oct 30 21:46:34.230878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.230812 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc"] Oct 30 21:46:34.231061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.231034 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" containerID="cri-o://75451fe2e11d20f17e7a56801d9202838ec1843d879ec90fd1088882ccb8696d" gracePeriod=30 Oct 30 21:46:34.234664 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.234638 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8"] Oct 30 21:46:34.234899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.234867 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" containerID="cri-o://8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3" gracePeriod=30 Oct 30 21:46:34.240171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.240148 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm"] Oct 30 21:46:34.240327 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.240310 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" containerID="cri-o://11684b25b18ae0e1cd7e5009064984fd859262a499019bf40eba80ad71d92bbc" gracePeriod=30 Oct 30 21:46:34.488364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.481259 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:46:34.491512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.491479 5039 generic.go:356] "Generic (PLEG): container finished" podID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerID="5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d" exitCode=0 Oct 30 21:46:34.491653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.491562 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" Oct 30 21:46:34.491721 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.491570 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" event={"ID":"192f7042-ada8-40d3-b5b1-5be2f8731c70","Type":"ContainerDied","Data":"5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d"} Oct 30 21:46:34.491779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.491717 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw" event={"ID":"192f7042-ada8-40d3-b5b1-5be2f8731c70","Type":"ContainerDied","Data":"2e317ea203fb9765b177c4e60aa4da8f478468795729acb084096352369b8775"} Oct 30 21:46:34.491779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.491746 5039 scope.go:122] "RemoveContainer" containerID="5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d" Oct 30 21:46:34.505090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505067 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505132 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505167 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505247 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505354 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505486 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505452 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505510 5039 scope.go:122] "RemoveContainer" containerID="5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d" Oct 30 21:46:34.505622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505550 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505667 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.505733 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.505702 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs\") pod \"192f7042-ada8-40d3-b5b1-5be2f8731c70\" (UID: \"192f7042-ada8-40d3-b5b1-5be2f8731c70\") " Oct 30 21:46:34.508084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.506223 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:34.508084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.506539 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:34.508084 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:46:34.506631 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d\": container with ID starting with 5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d not found: ID does not exist" containerID="5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d" Oct 30 21:46:34.508084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.506662 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d"} err="failed to get container status \"5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d\": rpc error: code = NotFound desc = could not find container \"5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d\": container with ID starting with 5dc623a6aadd4567866e943e739b8f02b2d3ea0c71eca2c5b212ed5a75db7d9d not found: ID does not exist" Oct 30 21:46:34.508084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.507273 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:34.508084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.507447 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:34.508866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.508845 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:34.510012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.509721 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "kube-api-access-qqbsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:34.510012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.509765 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:34.510012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.509929 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:34.511147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.511110 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo" pod "192f7042-ada8-40d3-b5b1-5be2f8731c70" (UID: "192f7042-ada8-40d3-b5b1-5be2f8731c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:34.623702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623604 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f7042-ada8-40d3-b5b1-5be2f8731c70-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.623702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623649 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.623702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623666 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbsh\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-kube-api-access-qqbsh\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.623702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623678 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.623702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623693 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.623702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623704 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.624155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623716 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f7042-ada8-40d3-b5b1-5be2f8731c70-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.624155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623725 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.624155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.623737 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f7042-ada8-40d3-b5b1-5be2f8731c70-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:34.710039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.709878 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:34.850759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.850726 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw"] Oct 30 21:46:34.856508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.856484 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-qt5lw"] Oct 30 21:46:34.929001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.928966 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" path="/var/lib/kubelet/pods/192f7042-ada8-40d3-b5b1-5be2f8731c70/volumes" Oct 30 21:46:34.937718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:34.937683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:35.434424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:35.432271 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:37.087061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:37.087023 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:37.938725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:37.938694 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:38.126506 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:38.126468 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:39.025382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.025346 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:39.519685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.519323 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:46:39.532645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.532588 5039 generic.go:356] "Generic (PLEG): container finished" podID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerID="8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3" exitCode=0 Oct 30 21:46:39.532795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.532780 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" Oct 30 21:46:39.532931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.532914 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" event={"ID":"b3f61f7a-3564-46b5-847e-63b642dadf42","Type":"ContainerDied","Data":"8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3"} Oct 30 21:46:39.533007 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.532992 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8" event={"ID":"b3f61f7a-3564-46b5-847e-63b642dadf42","Type":"ContainerDied","Data":"4b8dfab6a008c78fdd505c548b858ebaae504ad5fbd31d237a215ad09237a995"} Oct 30 21:46:39.533064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.533028 5039 scope.go:122] "RemoveContainer" containerID="8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3" Oct 30 21:46:39.545062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.544038 5039 generic.go:356] "Generic (PLEG): container finished" podID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerID="7c0b415c0271672a13ac9b33cea2afa91a2673182ccaeeb170890f2db6125e05" exitCode=0 Oct 30 21:46:39.545062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.544216 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" event={"ID":"51e92786-2a34-4fc8-8f8d-da19ee087997","Type":"ContainerDied","Data":"7c0b415c0271672a13ac9b33cea2afa91a2673182ccaeeb170890f2db6125e05"} Oct 30 21:46:39.555049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.553805 5039 generic.go:356] "Generic (PLEG): container finished" podID="303fe40a-e370-4a57-943f-a2be80c97f30" containerID="75451fe2e11d20f17e7a56801d9202838ec1843d879ec90fd1088882ccb8696d" exitCode=0 Oct 30 21:46:39.555049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.553881 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" event={"ID":"303fe40a-e370-4a57-943f-a2be80c97f30","Type":"ContainerDied","Data":"75451fe2e11d20f17e7a56801d9202838ec1843d879ec90fd1088882ccb8696d"} Oct 30 21:46:39.560488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.560453 5039 generic.go:356] "Generic (PLEG): container finished" podID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerID="11684b25b18ae0e1cd7e5009064984fd859262a499019bf40eba80ad71d92bbc" exitCode=0 Oct 30 21:46:39.560617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.560517 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" event={"ID":"8a1b2d63-7d92-4eac-9d71-46038b002151","Type":"ContainerDied","Data":"11684b25b18ae0e1cd7e5009064984fd859262a499019bf40eba80ad71d92bbc"} Oct 30 21:46:39.561216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.561198 5039 scope.go:122] "RemoveContainer" containerID="8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3" Oct 30 21:46:39.562564 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:46:39.562421 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3\": container with ID starting with 8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3 not found: ID does not exist" containerID="8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3" Oct 30 21:46:39.562564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.562457 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3"} err="failed to get container status \"8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3\": rpc error: code = NotFound desc = could not find container \"8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3\": container with ID starting with 8ccbedbecb8f46de8c6654880bde14f827aa7b0726349038d2dd4efe62c524a3 not found: ID does not exist" Oct 30 21:46:39.565831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.565811 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:46:39.585708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.585151 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.585708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.585204 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.585708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.585249 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.585985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.585300 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.586122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.586107 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.586193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.586163 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.586246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.586190 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.586246 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.586237 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.586344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.586287 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy\") pod \"b3f61f7a-3564-46b5-847e-63b642dadf42\" (UID: \"b3f61f7a-3564-46b5-847e-63b642dadf42\") " Oct 30 21:46:39.587741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.587712 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.587860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.587842 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.588021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.587996 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:39.588124 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.588057 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.588289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.588268 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.589602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.589421 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:39.590096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.590062 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "kube-api-access-2nllk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.593539 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.591479 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.595547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.595521 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token" pod "b3f61f7a-3564-46b5-847e-63b642dadf42" (UID: "b3f61f7a-3564-46b5-847e-63b642dadf42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.635896 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.635803 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:46:39.637815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.637790 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:46:39.688006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.687977 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.688145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688012 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.688145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688060 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.688145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688083 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.688145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688112 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.688145 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688140 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688180 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688231 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688272 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688314 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688340 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688406 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688424 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688438 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688532 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688608 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688648 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.688704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688649 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688718 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688756 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688782 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688847 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688865 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688876 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688906 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688945 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.688979 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.689011 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo\") pod \"51e92786-2a34-4fc8-8f8d-da19ee087997\" (UID: \"51e92786-2a34-4fc8-8f8d-da19ee087997\") " Oct 30 21:46:39.689430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.689067 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy\") pod \"8a1b2d63-7d92-4eac-9d71-46038b002151\" (UID: \"8a1b2d63-7d92-4eac-9d71-46038b002151\") " Oct 30 21:46:39.690465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.690425 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.690703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.690675 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.690969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.690818 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:39.690969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.690860 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.691106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.691087 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "kube-api-access-lf5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.691563 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.691832 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.691849 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692020 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692063 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692199 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692375 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.692724 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692588 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.693209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692789 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.693209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.692824 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.693209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.693057 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert\") pod \"303fe40a-e370-4a57-943f-a2be80c97f30\" (UID: \"303fe40a-e370-4a57-943f-a2be80c97f30\") " Oct 30 21:46:39.694171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694000 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "kube-api-access-ws6zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.694171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694044 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.694171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694137 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "kube-api-access-4l7xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.694729 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694704 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:39.694830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694769 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.694830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694799 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.694830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694820 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.694830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694818 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694832 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f61f7a-3564-46b5-847e-63b642dadf42-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694844 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694855 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694868 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694881 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694893 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1b2d63-7d92-4eac-9d71-46038b002151-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694906 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6zf\" (UniqueName: \"kubernetes.io/projected/51e92786-2a34-4fc8-8f8d-da19ee087997-kube-api-access-ws6zf\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694919 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694931 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694946 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694957 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694969 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694981 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.694993 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695005 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l7xk\" (UniqueName: \"kubernetes.io/projected/8a1b2d63-7d92-4eac-9d71-46038b002151-kube-api-access-4l7xk\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695018 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695030 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695042 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f61f7a-3564-46b5-847e-63b642dadf42-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695055 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e92786-2a34-4fc8-8f8d-da19ee087997-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695066 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695077 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695089 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695101 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695114 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695127 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5g2\" (UniqueName: \"kubernetes.io/projected/303fe40a-e370-4a57-943f-a2be80c97f30-kube-api-access-lf5g2\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695140 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.695642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.695153 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nllk\" (UniqueName: \"kubernetes.io/projected/b3f61f7a-3564-46b5-847e-63b642dadf42-kube-api-access-2nllk\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.696312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.696218 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:39.696312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.696248 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy" pod "8a1b2d63-7d92-4eac-9d71-46038b002151" (UID: "8a1b2d63-7d92-4eac-9d71-46038b002151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.696312 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.696277 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy" pod "51e92786-2a34-4fc8-8f8d-da19ee087997" (UID: "51e92786-2a34-4fc8-8f8d-da19ee087997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.697063 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.697037 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy" pod "303fe40a-e370-4a57-943f-a2be80c97f30" (UID: "303fe40a-e370-4a57-943f-a2be80c97f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:39.697642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.697619 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:39.796822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.796758 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303fe40a-e370-4a57-943f-a2be80c97f30-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.796822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.796785 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303fe40a-e370-4a57-943f-a2be80c97f30-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.796822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.796795 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.796822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.796803 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e92786-2a34-4fc8-8f8d-da19ee087997-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.796822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.796814 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1b2d63-7d92-4eac-9d71-46038b002151-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.796822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.796821 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303fe40a-e370-4a57-943f-a2be80c97f30-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.859842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.859803 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:39.877536 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.877513 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8"] Oct 30 21:46:39.882629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:39.882607 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b6tg8"] Oct 30 21:46:40.568893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.568818 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" Oct 30 21:46:40.569334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.568817 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc" event={"ID":"303fe40a-e370-4a57-943f-a2be80c97f30","Type":"ContainerDied","Data":"022b974b4a9daf5c20997a1e07ed078adf8bd241f499f6e5de1afefbfa190396"} Oct 30 21:46:40.569334 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.568964 5039 scope.go:122] "RemoveContainer" containerID="75451fe2e11d20f17e7a56801d9202838ec1843d879ec90fd1088882ccb8696d" Oct 30 21:46:40.570479 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.570456 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" Oct 30 21:46:40.570605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.570486 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm" event={"ID":"8a1b2d63-7d92-4eac-9d71-46038b002151","Type":"ContainerDied","Data":"bdb4f48a056bdf87c42ae66cbab7bec61774dd8e42748f9422833f5965948348"} Oct 30 21:46:40.573403 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.573375 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" Oct 30 21:46:40.573521 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.573450 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l" event={"ID":"51e92786-2a34-4fc8-8f8d-da19ee087997","Type":"ContainerDied","Data":"0c8459ac772281fcc129b8f60526b3071b95dbb535765ed60b9ed82ca4617a0e"} Oct 30 21:46:40.581418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.581373 5039 scope.go:122] "RemoveContainer" containerID="11684b25b18ae0e1cd7e5009064984fd859262a499019bf40eba80ad71d92bbc" Oct 30 21:46:40.591267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.591244 5039 scope.go:122] "RemoveContainer" containerID="7c0b415c0271672a13ac9b33cea2afa91a2673182ccaeeb170890f2db6125e05" Oct 30 21:46:40.623174 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.623149 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc"] Oct 30 21:46:40.632758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.632735 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-tm4pc"] Oct 30 21:46:40.661052 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.661023 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l"] Oct 30 21:46:40.666956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.666935 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-zhg9l"] Oct 30 21:46:40.696149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.696130 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm"] Oct 30 21:46:40.701737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.701716 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-f6kkm"] Oct 30 21:46:40.922305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.922270 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:40.923801 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.923771 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" path="/var/lib/kubelet/pods/303fe40a-e370-4a57-943f-a2be80c97f30/volumes" Oct 30 21:46:40.927344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.926098 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" path="/var/lib/kubelet/pods/51e92786-2a34-4fc8-8f8d-da19ee087997/volumes" Oct 30 21:46:40.927344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.926938 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" path="/var/lib/kubelet/pods/8a1b2d63-7d92-4eac-9d71-46038b002151/volumes" Oct 30 21:46:40.927807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:40.927746 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" path="/var/lib/kubelet/pods/b3f61f7a-3564-46b5-847e-63b642dadf42/volumes" Oct 30 21:46:42.598622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:42.598479 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:44.226126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:44.226092 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc"] Oct 30 21:46:44.226546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:44.226258 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" containerID="cri-o://71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d" gracePeriod=30 Oct 30 21:46:44.320244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:44.320211 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:45.898605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:45.898233 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.728966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:46.728925 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb"] Oct 30 21:46:46.729188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:46.729147 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" containerID="cri-o://078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec" gracePeriod=30 Oct 30 21:46:47.451569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:47.451534 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:48.062454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:48.062419 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-ggsbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:48.062623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:48.062485 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:49.110054 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.110015 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:49.500418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.500369 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:46:49.612232 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612203 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612299 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612332 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612363 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612435 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612513 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612516 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:49.612601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612542 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612640 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612702 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs\") pod \"e652f019-d1d9-4095-a57b-22d4b2016841\" (UID: \"e652f019-d1d9-4095-a57b-22d4b2016841\") " Oct 30 21:46:49.612924 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.612816 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:49.613365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.613099 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:49.613365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.613233 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.613365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.613251 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.613365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.613265 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e652f019-d1d9-4095-a57b-22d4b2016841-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.613365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.613262 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:49.613365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.613320 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:49.615087 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.615049 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:49.615305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.615265 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:49.615455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.615333 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:49.616139 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.616111 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8" pod "e652f019-d1d9-4095-a57b-22d4b2016841" (UID: "e652f019-d1d9-4095-a57b-22d4b2016841"). InnerVolumeSpecName "kube-api-access-9kjq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:49.638078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.638056 5039 generic.go:356] "Generic (PLEG): container finished" podID="e652f019-d1d9-4095-a57b-22d4b2016841" containerID="71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d" exitCode=0 Oct 30 21:46:49.638179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.638105 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" Oct 30 21:46:49.638179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.638147 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" event={"ID":"e652f019-d1d9-4095-a57b-22d4b2016841","Type":"ContainerDied","Data":"71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d"} Oct 30 21:46:49.638266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.638209 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc" event={"ID":"e652f019-d1d9-4095-a57b-22d4b2016841","Type":"ContainerDied","Data":"d625ce102ca9dbb5b4e79e88123fb36e9553cfa84a65b4b851e7b7cc1fcd3be5"} Oct 30 21:46:49.638266 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.638239 5039 scope.go:122] "RemoveContainer" containerID="71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d" Oct 30 21:46:49.650752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.650727 5039 scope.go:122] "RemoveContainer" containerID="71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d" Oct 30 21:46:49.651027 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:46:49.651010 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d\": container with ID starting with 71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d not found: ID does not exist" containerID="71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d" Oct 30 21:46:49.651100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.651033 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d"} err="failed to get container status \"71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d\": rpc error: code = NotFound desc = could not find container \"71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d\": container with ID starting with 71d4b2a12f216e5b2164bdae2768168d4c9bd72ce80664d43ec3003a7f85e94d not found: ID does not exist" Oct 30 21:46:49.700121 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.700097 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc"] Oct 30 21:46:49.706206 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.706186 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-ggsbc"] Oct 30 21:46:49.715899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.715872 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.715899 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.715900 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.716090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.715912 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.716090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.715924 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjq8\" (UniqueName: \"kubernetes.io/projected/e652f019-d1d9-4095-a57b-22d4b2016841-kube-api-access-9kjq8\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.716090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.715939 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e652f019-d1d9-4095-a57b-22d4b2016841-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:49.716090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:49.715952 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e652f019-d1d9-4095-a57b-22d4b2016841-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:50.024048 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:50.023967 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4"] Oct 30 21:46:50.116840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:50.115261 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:50.434844 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:50.434805 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8"] Oct 30 21:46:50.500630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:50.500598 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:50.888823 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:50.888795 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:50.921527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:50.921500 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" path="/var/lib/kubelet/pods/e652f019-d1d9-4095-a57b-22d4b2016841/volumes" Oct 30 21:46:51.467891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:51.467853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:52.000433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.000362 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:46:52.060272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060239 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060363 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060423 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060490 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060521 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060556 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060641 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060685 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060733 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert\") pod \"65e6f922-b778-47b5-ad07-62f41980f30f\" (UID: \"65e6f922-b778-47b5-ad07-62f41980f30f\") " Oct 30 21:46:52.060778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.060752 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:52.061427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.061232 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:52.061551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.061367 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:52.061620 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.061566 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:52.061754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.061726 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:52.064125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.062995 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.064125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.063020 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.064125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.063036 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e6f922-b778-47b5-ad07-62f41980f30f-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.064125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.063051 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.064125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.063062 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.064654 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.064633 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:52.064774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.064684 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:52.064889 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.064772 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:52.064889 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.064868 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m" pod "65e6f922-b778-47b5-ad07-62f41980f30f" (UID: "65e6f922-b778-47b5-ad07-62f41980f30f"). InnerVolumeSpecName "kube-api-access-7jp2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:52.163795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.163766 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e6f922-b778-47b5-ad07-62f41980f30f-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.163795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.163792 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e6f922-b778-47b5-ad07-62f41980f30f-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.163950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.163804 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.163950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.163825 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jp2m\" (UniqueName: \"kubernetes.io/projected/65e6f922-b778-47b5-ad07-62f41980f30f-kube-api-access-7jp2m\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.524496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.524457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:52.670513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.670475 5039 generic.go:356] "Generic (PLEG): container finished" podID="65e6f922-b778-47b5-ad07-62f41980f30f" containerID="078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec" exitCode=0 Oct 30 21:46:52.670513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.670520 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" Oct 30 21:46:52.670757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.670559 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" event={"ID":"65e6f922-b778-47b5-ad07-62f41980f30f","Type":"ContainerDied","Data":"078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec"} Oct 30 21:46:52.670757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.670616 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb" event={"ID":"65e6f922-b778-47b5-ad07-62f41980f30f","Type":"ContainerDied","Data":"21b53beeef3a155cf4570a52712af627e87c69f9a84ff1ff753e48fe0ff64dde"} Oct 30 21:46:52.670757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.670674 5039 scope.go:122] "RemoveContainer" containerID="078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec" Oct 30 21:46:52.681208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.681189 5039 scope.go:122] "RemoveContainer" containerID="078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec" Oct 30 21:46:52.681488 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:46:52.681465 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec\": container with ID starting with 078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec not found: ID does not exist" containerID="078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec" Oct 30 21:46:52.681551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.681502 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec"} err="failed to get container status \"078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec\": rpc error: code = NotFound desc = could not find container \"078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec\": container with ID starting with 078cf1e201e5864b14a35453047bfccdae8f1d448e24d72cbb4c391cb279b7ec not found: ID does not exist" Oct 30 21:46:52.743453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.743431 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb"] Oct 30 21:46:52.746889 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.746868 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4bcwb"] Oct 30 21:46:52.921855 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:52.921830 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" path="/var/lib/kubelet/pods/65e6f922-b778-47b5-ad07-62f41980f30f/volumes" Oct 30 21:46:54.204212 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:54.203940 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:55.184761 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:46:55.184728 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:46:55.370858 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.370825 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:55.747376 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747346 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh"] Oct 30 21:46:55.747903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747881 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.747903 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747901 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747913 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747920 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747942 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747948 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747965 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747971 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747985 5039 cpu_manager.go:414] "containerMap: removing container" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.747991 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748008 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748013 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748032 5039 cpu_manager.go:414] "containerMap: removing container" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748038 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748050 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748056 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748066 5039 cpu_manager.go:414] "containerMap: removing container" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748073 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748084 5039 cpu_manager.go:414] "containerMap: removing container" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748091 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748105 5039 cpu_manager.go:414] "containerMap: removing container" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748113 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748121 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748128 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:46:55.748130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748140 5039 cpu_manager.go:414] "containerMap: removing container" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748147 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748156 5039 cpu_manager.go:414] "containerMap: removing container" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748163 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748181 5039 cpu_manager.go:414] "containerMap: removing container" podUID="1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" containerName="main" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748189 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" containerName="main" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748200 5039 cpu_manager.go:414] "containerMap: removing container" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748207 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748218 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748225 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748235 5039 cpu_manager.go:414] "containerMap: removing container" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748241 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748254 5039 cpu_manager.go:414] "containerMap: removing container" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748261 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748275 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748282 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748293 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748300 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748310 5039 cpu_manager.go:414] "containerMap: removing container" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748317 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748484 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f61f7a-3564-46b5-847e-63b642dadf42" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748501 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748543 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748555 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748563 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="192f7042-ada8-40d3-b5b1-5be2f8731c70" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748572 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748581 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748588 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f69bd36-dade-487d-bba3-db4ce7669a44" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748601 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9227f671-9a58-4202-a58a-8338b7c044da" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748611 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748620 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8dbc5be-53dd-430e-ab1b-bea41e346093" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748630 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e2278c-830b-42e1-b81c-aae104bffa53" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748643 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e92786-2a34-4fc8-8f8d-da19ee087997" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748651 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748660 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1b2d63-7d92-4eac-9d71-46038b002151" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748667 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e6f922-b778-47b5-ad07-62f41980f30f" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748676 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="303fe40a-e370-4a57-943f-a2be80c97f30" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748684 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c52fd18-675a-4b47-a3ff-b7b4199bd5e3" containerName="main" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748694 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="e652f019-d1d9-4095-a57b-22d4b2016841" containerName="istio-proxy" Oct 30 21:46:55.749194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.748703 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:46:55.763242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.763208 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.789452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.789432 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3d5d13-openshift-default-dockercfg-tgbxm\"" Oct 30 21:46:55.800869 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.800843 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh"] Oct 30 21:46:55.801592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801569 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.801695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801610 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.801695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801637 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjf7v\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.801813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801728 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.801870 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801833 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.801927 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801902 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.801967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.801948 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.802046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.802031 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.802078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.802059 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.882666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.882626 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:55.903701 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.903671 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.903848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.903703 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.903848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.903801 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.903848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.903829 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.903854 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjf7v\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.903881 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904125 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904164 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904181 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904212 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904254 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904291 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.904784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.904756 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.905038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.905008 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.906489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.906470 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.907076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.907054 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.925005 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.924985 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:55.925104 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:55.925019 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjf7v\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v\") pod \"e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:56.071783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.071688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.080608 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.080580 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:56.222017 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.221979 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.414333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.414294 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh"] Oct 30 21:46:56.417424 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:46:56.415028 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3404db_e958_4f7c_b4a2_8edad0eab199.slice/crio-34d4f46d025f96b9fc0d9e6c04df40121c021f60707aaaef35faecb034fc9070 WatchSource:0}: Error finding container 34d4f46d025f96b9fc0d9e6c04df40121c021f60707aaaef35faecb034fc9070: Status 404 returned error can't find the container with id 34d4f46d025f96b9fc0d9e6c04df40121c021f60707aaaef35faecb034fc9070 Oct 30 21:46:56.421628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.421591 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:56.421735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.421677 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:56.421735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.421722 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:56.702618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.702536 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" event={"ID":"bb3404db-e958-4f7c-b4a2-8edad0eab199","Type":"ContainerStarted","Data":"4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb"} Oct 30 21:46:56.702618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.702593 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" event={"ID":"bb3404db-e958-4f7c-b4a2-8edad0eab199","Type":"ContainerStarted","Data":"34d4f46d025f96b9fc0d9e6c04df40121c021f60707aaaef35faecb034fc9070"} Oct 30 21:46:56.741450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:56.741385 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podStartSLOduration=1.7413692950000002 podStartE2EDuration="1.741369295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:56.740061324 +0000 UTC m=+2255.950000952" watchObservedRunningTime="2026-10-30 21:46:56.741369295 +0000 UTC m=+2255.951308918" Oct 30 21:46:57.080941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:57.080852 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:57.086156 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:57.086132 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:57.696756 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:57.696708 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:57.711208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:57.711179 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:57.711901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:57.711879 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:46:58.912726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:58.912678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:59.541485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:59.541451 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:59.703839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:46:59.703801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:00.257013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:00.256974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:00.415090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:00.415054 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:00.942646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:00.942616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.100308 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.100273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.116875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.116741 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h"] Oct 30 21:47:01.117422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.117384 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:47:01.117422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.117421 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:47:01.117581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.117551 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc669e4-6495-4c9a-ba02-d83ae7bcaa6e" containerName="cleanup" Oct 30 21:47:01.117581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.117565 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd05d0e-0b80-4802-81b7-38929ba42ef4" containerName="cleanup" Oct 30 21:47:01.117581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.117580 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b8025a-5df4-404f-b60c-54ff2b23a014" containerName="cleanup" Oct 30 21:47:01.121131 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.121109 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.154359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.154257 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h"] Oct 30 21:47:01.195267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.183898 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzj7p\" (UniqueName: \"kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.195267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.184013 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.195267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.184074 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.195267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.184222 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.223941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.215599 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c19461-serving-cert\"" Oct 30 21:47:01.288847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.286686 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzj7p\" (UniqueName: \"kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.288847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.286754 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.288847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.286785 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.288847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.286882 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.288847 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:01.287370 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-serving-cert: secret "maas-api-e2e-extmod-a-c19461-serving-cert" not found Oct 30 21:47:01.288847 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:01.287571 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls podName:4d128504-5584-4533-9f19-1c00d1c10c76 nodeName:}" failed. No retries permitted until 2026-10-30 21:47:01.787448268 +0000 UTC m=+2260.997387899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls") pod "maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" (UID: "4d128504-5584-4533-9f19-1c00d1c10c76") : secret "maas-api-e2e-extmod-a-c19461-serving-cert" not found Oct 30 21:47:01.295243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.291068 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.295475 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.291873 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.295623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.295571 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.312992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.312344 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzj7p\" (UniqueName: \"kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.439871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.439839 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.589132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.589041 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:01.792885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.792855 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:01.795061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:01.795030 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:02.038204 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:02.038173 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:02.350746 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:02.350719 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h"] Oct 30 21:47:02.352342 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:47:02.352311 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d128504_5584_4533_9f19_1c00d1c10c76.slice/crio-f3197ff5c598bbb5291567acd4dc62822dc418cab522b20b5600593229683262 WatchSource:0}: Error finding container f3197ff5c598bbb5291567acd4dc62822dc418cab522b20b5600593229683262: Status 404 returned error can't find the container with id f3197ff5c598bbb5291567acd4dc62822dc418cab522b20b5600593229683262 Oct 30 21:47:02.353942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:02.353921 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:47:02.745957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:02.745921 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" event={"ID":"4d128504-5584-4533-9f19-1c00d1c10c76","Type":"ContainerStarted","Data":"f3197ff5c598bbb5291567acd4dc62822dc418cab522b20b5600593229683262"} Oct 30 21:47:03.172445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:03.172367 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:03.472320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:03.472233 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh"] Oct 30 21:47:03.553591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:03.553555 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:03.754335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:03.754242 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" event={"ID":"4d128504-5584-4533-9f19-1c00d1c10c76","Type":"ContainerStarted","Data":"c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2"} Oct 30 21:47:03.754491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:03.754345 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:03.792565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:03.792512 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" podStartSLOduration=2.38246751 podStartE2EDuration="2.792498825s" totalImagesPullingTime="410.031315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:03.789592118 +0000 UTC m=+2262.999531747" watchObservedRunningTime="2026-10-30 21:47:03.792498825 +0000 UTC m=+2263.002438453" Oct 30 21:47:04.517137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:04.517106 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:04.804973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:04.804897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.044811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.044781 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" podUID="330af03b-8043-4376-8bf3-02f78c78047d" containerName="main" containerID="cri-o://e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9" gracePeriod=45 Oct 30 21:47:05.165893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.165864 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.310596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.310535 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:47:05.333604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.333575 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.378070 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378041 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home\") pod \"330af03b-8043-4376-8bf3-02f78c78047d\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " Oct 30 21:47:05.378183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378081 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm\") pod \"330af03b-8043-4376-8bf3-02f78c78047d\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " Oct 30 21:47:05.378183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378154 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs\" (UniqueName: \"kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs\") pod \"330af03b-8043-4376-8bf3-02f78c78047d\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " Oct 30 21:47:05.378321 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378306 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir\") pod \"330af03b-8043-4376-8bf3-02f78c78047d\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " Oct 30 21:47:05.378321 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378308 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home" pod "330af03b-8043-4376-8bf3-02f78c78047d" (UID: "330af03b-8043-4376-8bf3-02f78c78047d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.378496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378382 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5\" (UniqueName: \"kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5\") pod \"330af03b-8043-4376-8bf3-02f78c78047d\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " Oct 30 21:47:05.378496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378455 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache\") pod \"330af03b-8043-4376-8bf3-02f78c78047d\" (UID: \"330af03b-8043-4376-8bf3-02f78c78047d\") " Oct 30 21:47:05.378619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378597 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir" pod "330af03b-8043-4376-8bf3-02f78c78047d" (UID: "330af03b-8043-4376-8bf3-02f78c78047d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.378809 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.378789 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache" pod "330af03b-8043-4376-8bf3-02f78c78047d" (UID: "330af03b-8043-4376-8bf3-02f78c78047d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.379056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.379034 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.379147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.379061 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.379147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.379077 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.380450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.380426 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm" pod "330af03b-8043-4376-8bf3-02f78c78047d" (UID: "330af03b-8043-4376-8bf3-02f78c78047d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.380572 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.380547 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs" pod "330af03b-8043-4376-8bf3-02f78c78047d" (UID: "330af03b-8043-4376-8bf3-02f78c78047d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:05.383768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.383742 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5" pod "330af03b-8043-4376-8bf3-02f78c78047d" (UID: "330af03b-8043-4376-8bf3-02f78c78047d"). InnerVolumeSpecName "kube-api-access-fwnb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:05.456890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.456852 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" podUID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" containerName="main" containerID="cri-o://3cfa18234a6b9e251236a37aea0c555bb4ec2deb004cc2b8c2d3170a13f3f07c" gracePeriod=45 Oct 30 21:47:05.480184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.480157 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330af03b-8043-4376-8bf3-02f78c78047d-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.480272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.480185 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330af03b-8043-4376-8bf3-02f78c78047d-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.480272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.480199 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwnb5\" (UniqueName: \"kubernetes.io/projected/330af03b-8043-4376-8bf3-02f78c78047d-kube-api-access-fwnb5\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.487081 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.487049 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" podUID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": dial tcp 10.132.0.123:8000: connect: connection refused" Oct 30 21:47:05.545132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.545106 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w"] Oct 30 21:47:05.545956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.545818 5039 cpu_manager.go:414] "containerMap: removing container" podUID="330af03b-8043-4376-8bf3-02f78c78047d" containerName="main" Oct 30 21:47:05.545956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.545842 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330af03b-8043-4376-8bf3-02f78c78047d" containerName="main" Oct 30 21:47:05.546107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.546049 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="330af03b-8043-4376-8bf3-02f78c78047d" containerName="main" Oct 30 21:47:05.549936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.549915 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.571436 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.569822 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-427a6e-openshift-default-dockercfg-kkqzj\"" Oct 30 21:47:05.571436 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.571299 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w"] Oct 30 21:47:05.580831 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.580809 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zktj\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.580956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.580863 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.580959 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.581005 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.581049 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.581088 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.581137 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.581168 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.581241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.581198 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.642907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.642871 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711195 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711249 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711296 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711324 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711351 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711464 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zktj\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711514 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.716485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.711620 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.738199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.732021 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.752109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.752069 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.752614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.752588 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.753076 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.753054 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.761956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.759697 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.761956 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.759379 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.786317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.783663 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.789185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.789147 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.789839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.789810 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zktj\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.794255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.794229 5039 generic.go:356] "Generic (PLEG): container finished" podID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" containerID="3cfa18234a6b9e251236a37aea0c555bb4ec2deb004cc2b8c2d3170a13f3f07c" exitCode=0 Oct 30 21:47:05.794364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.794314 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" event={"ID":"e8109af0-b4d0-4eae-afda-e1e9eaa52721","Type":"ContainerDied","Data":"3cfa18234a6b9e251236a37aea0c555bb4ec2deb004cc2b8c2d3170a13f3f07c"} Oct 30 21:47:05.796348 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.796323 5039 generic.go:356] "Generic (PLEG): container finished" podID="330af03b-8043-4376-8bf3-02f78c78047d" containerID="e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9" exitCode=0 Oct 30 21:47:05.796528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.796512 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" event={"ID":"330af03b-8043-4376-8bf3-02f78c78047d","Type":"ContainerDied","Data":"e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9"} Oct 30 21:47:05.796605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.796545 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" event={"ID":"330af03b-8043-4376-8bf3-02f78c78047d","Type":"ContainerDied","Data":"646cb629a4d0f8f0ce3a7cc0653c5ca9629cdbe16af700a959c819483fccb160"} Oct 30 21:47:05.796605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.796565 5039 scope.go:122] "RemoveContainer" containerID="e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9" Oct 30 21:47:05.796736 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.796717 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4" Oct 30 21:47:05.829331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.820296 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.844108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.844072 5039 scope.go:122] "RemoveContainer" containerID="e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9" Oct 30 21:47:05.844538 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:05.844497 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9\": container with ID starting with e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9 not found: ID does not exist" containerID="e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9" Oct 30 21:47:05.844601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.844533 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9"} err="failed to get container status \"e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9\": rpc error: code = NotFound desc = could not find container \"e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9\": container with ID starting with e4a18b26917003c07f07cda4968cf6a38fd924484318724a2c30bb141a084ac9 not found: ID does not exist" Oct 30 21:47:05.845639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.845616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:05.855129 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.855106 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:47:05.859228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.859205 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4"] Oct 30 21:47:05.872856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.869864 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-e06b90/auth-test-model-e06b90-kserve-7cbbd694b9-lwrc4"] Oct 30 21:47:05.880626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.880602 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:05.963912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.963879 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache\") pod \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " Oct 30 21:47:05.964052 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964032 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir\") pod \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " Oct 30 21:47:05.964112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964093 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs\") pod \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " Oct 30 21:47:05.964193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964177 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm\") pod \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " Oct 30 21:47:05.964258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964189 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache" pod "e8109af0-b4d0-4eae-afda-e1e9eaa52721" (UID: "e8109af0-b4d0-4eae-afda-e1e9eaa52721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.964317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964286 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26\" (UniqueName: \"kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26\") pod \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " Oct 30 21:47:05.964317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964289 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir" pod "e8109af0-b4d0-4eae-afda-e1e9eaa52721" (UID: "e8109af0-b4d0-4eae-afda-e1e9eaa52721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.964428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964316 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home\") pod \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\" (UID: \"e8109af0-b4d0-4eae-afda-e1e9eaa52721\") " Oct 30 21:47:05.964584 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964567 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home" pod "e8109af0-b4d0-4eae-afda-e1e9eaa52721" (UID: "e8109af0-b4d0-4eae-afda-e1e9eaa52721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.964813 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964798 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.964875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964816 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.964875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.964824 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:05.967113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.967085 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm" pod "e8109af0-b4d0-4eae-afda-e1e9eaa52721" (UID: "e8109af0-b4d0-4eae-afda-e1e9eaa52721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:05.967113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.967100 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs" pod "e8109af0-b4d0-4eae-afda-e1e9eaa52721" (UID: "e8109af0-b4d0-4eae-afda-e1e9eaa52721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:05.967420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:05.967364 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26" pod "e8109af0-b4d0-4eae-afda-e1e9eaa52721" (UID: "e8109af0-b4d0-4eae-afda-e1e9eaa52721"). InnerVolumeSpecName "kube-api-access-skf26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:06.065742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.065704 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skf26\" (UniqueName: \"kubernetes.io/projected/e8109af0-b4d0-4eae-afda-e1e9eaa52721-kube-api-access-skf26\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:06.065742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.065737 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8109af0-b4d0-4eae-afda-e1e9eaa52721-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:06.065918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.065754 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8109af0-b4d0-4eae-afda-e1e9eaa52721-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:06.196504 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.196470 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:06.211573 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.211527 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w"] Oct 30 21:47:06.212351 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:47:06.212327 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e574d24_b86a_48cd_af75_0bd5651d0ca2.slice/crio-b2eaf31132eb38310129edbcfa36b48903a093170e9cc5a007f204db15974038 WatchSource:0}: Error finding container b2eaf31132eb38310129edbcfa36b48903a093170e9cc5a007f204db15974038: Status 404 returned error can't find the container with id b2eaf31132eb38310129edbcfa36b48903a093170e9cc5a007f204db15974038 Oct 30 21:47:06.215439 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.215377 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:06.215523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.215475 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:06.215523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.215502 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:06.413673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.413178 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:06.804919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.804888 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" Oct 30 21:47:06.805422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.804884 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8" event={"ID":"e8109af0-b4d0-4eae-afda-e1e9eaa52721","Type":"ContainerDied","Data":"649e6e894e9183b3a8c6107c3852506aebe1eb048e57a4b8f46f17ac39056b05"} Oct 30 21:47:06.805422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.805020 5039 scope.go:122] "RemoveContainer" containerID="3cfa18234a6b9e251236a37aea0c555bb4ec2deb004cc2b8c2d3170a13f3f07c" Oct 30 21:47:06.807837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.807803 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" event={"ID":"8e574d24-b86a-48cd-af75-0bd5651d0ca2","Type":"ContainerStarted","Data":"91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617"} Oct 30 21:47:06.807969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.807879 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" event={"ID":"8e574d24-b86a-48cd-af75-0bd5651d0ca2","Type":"ContainerStarted","Data":"b2eaf31132eb38310129edbcfa36b48903a093170e9cc5a007f204db15974038"} Oct 30 21:47:06.855011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.854988 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8"] Oct 30 21:47:06.864653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.864632 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3172e5/auth-test-model-3172e5-kserve-6c4ddf48fc-z4xs8"] Oct 30 21:47:06.881074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.881055 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:06.905719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.905677 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podStartSLOduration=1.905665277 podStartE2EDuration="1.905665277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:06.90098769 +0000 UTC m=+2266.110927318" watchObservedRunningTime="2026-10-30 21:47:06.905665277 +0000 UTC m=+2266.115604905" Oct 30 21:47:06.921871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.921847 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330af03b-8043-4376-8bf3-02f78c78047d" path="/var/lib/kubelet/pods/330af03b-8043-4376-8bf3-02f78c78047d/volumes" Oct 30 21:47:06.922290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:06.922277 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" path="/var/lib/kubelet/pods/e8109af0-b4d0-4eae-afda-e1e9eaa52721/volumes" Oct 30 21:47:07.238651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:07.238601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:07.886064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:07.886032 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:08.820888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:08.820862 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:08.821890 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:08.821869 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:47:08.889525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:08.889497 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:47:09.033151 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:09.033119 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:09.491551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:09.491521 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:09.720011 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:09.719953 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:09.770252 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:09.770199 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 21:47:11.290783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:11.290752 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:11.602830 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:11.602741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.489590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:13.489554 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.011377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:15.011343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:15.261752 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:15.261664 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:16.937501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:16.937464 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.493218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.493183 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" podUID="d9d8562c-8b20-4c70-81f2-7a4433a63ee1" containerName="main" containerID="cri-o://bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7" gracePeriod=45 Oct 30 21:47:18.756341 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.756244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.765524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.765499 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:47:18.861610 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.861579 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs\") pod \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " Oct 30 21:47:18.861780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.861692 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm\") pod \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " Oct 30 21:47:18.861780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.861734 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs\" (UniqueName: \"kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs\") pod \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " Oct 30 21:47:18.861780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.861771 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home\") pod \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " Oct 30 21:47:18.861967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.861798 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache\") pod \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " Oct 30 21:47:18.861967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.861864 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir\") pod \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\" (UID: \"d9d8562c-8b20-4c70-81f2-7a4433a63ee1\") " Oct 30 21:47:18.862068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.862046 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home" pod "d9d8562c-8b20-4c70-81f2-7a4433a63ee1" (UID: "d9d8562c-8b20-4c70-81f2-7a4433a63ee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:18.862141 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.862089 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache" pod "d9d8562c-8b20-4c70-81f2-7a4433a63ee1" (UID: "d9d8562c-8b20-4c70-81f2-7a4433a63ee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:18.862290 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.862268 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir" pod "d9d8562c-8b20-4c70-81f2-7a4433a63ee1" (UID: "d9d8562c-8b20-4c70-81f2-7a4433a63ee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:18.862605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.862579 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:18.862689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.862609 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:18.862689 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.862627 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:18.863743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.863725 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs" pod "d9d8562c-8b20-4c70-81f2-7a4433a63ee1" (UID: "d9d8562c-8b20-4c70-81f2-7a4433a63ee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:18.863965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.863940 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm" pod "d9d8562c-8b20-4c70-81f2-7a4433a63ee1" (UID: "d9d8562c-8b20-4c70-81f2-7a4433a63ee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:18.864077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.863981 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs" pod "d9d8562c-8b20-4c70-81f2-7a4433a63ee1" (UID: "d9d8562c-8b20-4c70-81f2-7a4433a63ee1"). InnerVolumeSpecName "kube-api-access-l96zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:18.884477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.884453 5039 generic.go:356] "Generic (PLEG): container finished" podID="d9d8562c-8b20-4c70-81f2-7a4433a63ee1" containerID="bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7" exitCode=0 Oct 30 21:47:18.884580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.884501 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" Oct 30 21:47:18.884580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.884518 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" event={"ID":"d9d8562c-8b20-4c70-81f2-7a4433a63ee1","Type":"ContainerDied","Data":"bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7"} Oct 30 21:47:18.884580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.884564 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh" event={"ID":"d9d8562c-8b20-4c70-81f2-7a4433a63ee1","Type":"ContainerDied","Data":"e9d0625fe525e99e8c879b0311412315bb31df8ef185816c4ec8fc716e27a673"} Oct 30 21:47:18.884726 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.884582 5039 scope.go:122] "RemoveContainer" containerID="bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7" Oct 30 21:47:18.895176 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.895158 5039 scope.go:122] "RemoveContainer" containerID="bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7" Oct 30 21:47:18.895492 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:18.895472 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7\": container with ID starting with bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7 not found: ID does not exist" containerID="bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7" Oct 30 21:47:18.895579 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.895502 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7"} err="failed to get container status \"bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7\": rpc error: code = NotFound desc = could not find container \"bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7\": container with ID starting with bfca6e8d7306dc11e6f2df489ff480f33e504c7cc5d6d34a480a6ef0b963f5f7 not found: ID does not exist" Oct 30 21:47:18.932423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.932382 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh"] Oct 30 21:47:18.941128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.941104 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f49db2/auth-test-model-f49db2-kserve-5cfc58d5cc-ckszh"] Oct 30 21:47:18.963648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.963626 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:18.963735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.963649 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96zs\" (UniqueName: \"kubernetes.io/projected/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-kube-api-access-l96zs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:18.963735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:18.963660 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d8562c-8b20-4c70-81f2-7a4433a63ee1-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:20.464501 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:20.464457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.706940 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:20.706903 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.934958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:20.934917 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d8562c-8b20-4c70-81f2-7a4433a63ee1" path="/var/lib/kubelet/pods/d9d8562c-8b20-4c70-81f2-7a4433a63ee1/volumes" Oct 30 21:47:22.295865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:22.295831 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:24.058520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:24.058481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:25.184313 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:25.184276 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:47:25.839480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:25.839439 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:27.342061 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.342021 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g"] Oct 30 21:47:27.342769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.342748 5039 cpu_manager.go:414] "containerMap: removing container" podUID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" containerName="main" Oct 30 21:47:27.342880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.342771 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" containerName="main" Oct 30 21:47:27.342880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.342820 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d9d8562c-8b20-4c70-81f2-7a4433a63ee1" containerName="main" Oct 30 21:47:27.342880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.342829 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d8562c-8b20-4c70-81f2-7a4433a63ee1" containerName="main" Oct 30 21:47:27.343026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.343000 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8109af0-b4d0-4eae-afda-e1e9eaa52721" containerName="main" Oct 30 21:47:27.343026 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.343017 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d8562c-8b20-4c70-81f2-7a4433a63ee1" containerName="main" Oct 30 21:47:27.348036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.348016 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.374979 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.374949 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g"] Oct 30 21:47:27.379116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.379095 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3d5d13-serving-cert\"" Oct 30 21:47:27.455413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.455368 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqzf\" (UniqueName: \"kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.455530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.455484 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.455530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.455521 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.455671 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.455654 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.487468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.487446 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:27.556995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.556974 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqzf\" (UniqueName: \"kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.557093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.557011 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.557093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.557041 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.557182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.557136 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.557820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.557796 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.559513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.559491 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.559609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.559591 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.574480 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.574452 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqzf\" (UniqueName: \"kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf\") pod \"maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.660914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.660878 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:27.675440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.675385 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:27.965282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:27.965251 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g"] Oct 30 21:47:27.966179 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:47:27.966152 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cc7de8_a788_4d88_a3f0_d14270a16b7b.slice/crio-f2244cc753a2a158cc1f02ea589c996dba66a78fc876ed385a165c09ea8ab9f4 WatchSource:0}: Error finding container f2244cc753a2a158cc1f02ea589c996dba66a78fc876ed385a165c09ea8ab9f4: Status 404 returned error can't find the container with id f2244cc753a2a158cc1f02ea589c996dba66a78fc876ed385a165c09ea8ab9f4 Oct 30 21:47:28.945826 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:28.945792 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" event={"ID":"f3cc7de8-a788-4d88-a3f0-d14270a16b7b","Type":"ContainerStarted","Data":"0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604"} Oct 30 21:47:28.946203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:28.945841 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" event={"ID":"f3cc7de8-a788-4d88-a3f0-d14270a16b7b","Type":"ContainerStarted","Data":"f2244cc753a2a158cc1f02ea589c996dba66a78fc876ed385a165c09ea8ab9f4"} Oct 30 21:47:28.946203 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:28.945959 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:28.981283 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:28.981238 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" podStartSLOduration=1.674502038 podStartE2EDuration="1.981227712s" totalImagesPullingTime="306.725674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:28.98085972 +0000 UTC m=+2288.190799350" watchObservedRunningTime="2026-10-30 21:47:28.981227712 +0000 UTC m=+2288.191167340" Oct 30 21:47:29.327465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:29.327351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:30.953234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:30.953195 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.651586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.651513 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v"] Oct 30 21:47:32.655761 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.655743 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.678006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.677982 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3bc8ea-serving-cert\"" Oct 30 21:47:32.693102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.693075 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v"] Oct 30 21:47:32.729884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.729856 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.730001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.729918 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.730049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.730034 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.730157 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.730139 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ct6\" (UniqueName: \"kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.830790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.830763 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.830912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.830832 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ct6\" (UniqueName: \"kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.830912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.830870 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.830912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.830908 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.831687 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.831665 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.833130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.833101 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.833221 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.833133 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.849020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.848998 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ct6\" (UniqueName: \"kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6\") pod \"maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:32.887708 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.887673 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.970217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:32.970121 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:33.160530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:33.160491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:33.281525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:33.281465 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v"] Oct 30 21:47:33.285359 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:47:33.285333 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b0633b_746d_4660_ac74_5582983f60f4.slice/crio-da2f09085f30cfb4ed545dc7d139a828a5f8e29ff12c1a65bca8cd68271e7ace WatchSource:0}: Error finding container da2f09085f30cfb4ed545dc7d139a828a5f8e29ff12c1a65bca8cd68271e7ace: Status 404 returned error can't find the container with id da2f09085f30cfb4ed545dc7d139a828a5f8e29ff12c1a65bca8cd68271e7ace Oct 30 21:47:33.983443 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:33.983410 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" event={"ID":"f1b0633b-746d-4660-ac74-5582983f60f4","Type":"ContainerStarted","Data":"21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06"} Oct 30 21:47:33.983839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:33.983466 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" event={"ID":"f1b0633b-746d-4660-ac74-5582983f60f4","Type":"ContainerStarted","Data":"da2f09085f30cfb4ed545dc7d139a828a5f8e29ff12c1a65bca8cd68271e7ace"} Oct 30 21:47:33.983839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:33.983490 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:34.093416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:34.092912 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:34.292615 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:34.292533 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:34.959136 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:34.959112 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 21:47:35.012596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:35.012548 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" podStartSLOduration=2.727237128 podStartE2EDuration="3.012533513s" totalImagesPullingTime="285.296385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:34.064144103 +0000 UTC m=+2293.274083733" watchObservedRunningTime="2026-10-30 21:47:35.012533513 +0000 UTC m=+2294.222473136" Oct 30 21:47:35.056907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:35.056869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.489332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:35.489297 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.197441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:36.194251 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:36.599545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:36.599468 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:37.547695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:37.547650 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.179884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:39.179851 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld"] Oct 30 21:47:39.180259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:39.180084 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" containerID="cri-o://2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5" gracePeriod=30 Oct 30 21:47:39.187697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:39.187668 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4"] Oct 30 21:47:39.187907 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:39.187873 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" containerID="cri-o://6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a" gracePeriod=30 Oct 30 21:47:39.810343 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:39.810309 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.996003 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:39.995977 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:47:40.459988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:40.459786 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.264078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:41.264040 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.496731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:41.496696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:41.884166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:41.884135 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.118982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:42.118946 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.432093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:42.432059 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.788966 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:42.788878 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.007179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:43.007148 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.847678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:43.847637 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:44.296194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.296158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:44.479992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.479968 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:47:44.482808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.482790 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:47:44.518945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.518923 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.518972 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519011 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519069 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519115 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519180 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519219 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519282 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519314 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519342 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519409 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.519563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519426 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519465 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519498 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519529 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519593 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519614 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.519811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519630 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo\") pod \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\" (UID: \"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d\") " Oct 30 21:47:44.519811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519628 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.519811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519696 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.519811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519753 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy\") pod \"570a2609-dedf-4228-9fff-3d18d558f715\" (UID: \"570a2609-dedf-4228-9fff-3d18d558f715\") " Oct 30 21:47:44.520122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519818 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.520122 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.519858 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.520231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.520115 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:44.520525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.520496 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:44.521107 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521086 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.521248 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521227 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.521351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521337 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521145 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521367 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521374 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521381 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.521896 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "kube-api-access-4xs45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.522518 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "kube-api-access-pts27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.522644 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.522658 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.522683 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.522696 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a2609-dedf-4228-9fff-3d18d558f715-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.522769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.522738 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:44.523330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.523223 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:44.523378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.523339 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:44.523581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.523558 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token" pod "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" (UID: "2c9a100a-efbc-4fe2-a74a-c12ffbfb326d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:44.523650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.523589 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy" pod "570a2609-dedf-4228-9fff-3d18d558f715" (UID: "570a2609-dedf-4228-9fff-3d18d558f715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:44.624089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624059 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xs45\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-kube-api-access-4xs45\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624087 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624101 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pts27\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-kube-api-access-pts27\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624112 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624123 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a2609-dedf-4228-9fff-3d18d558f715-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624134 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624146 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a2609-dedf-4228-9fff-3d18d558f715-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624159 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624170 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624181 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.624277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.624192 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a2609-dedf-4228-9fff-3d18d558f715-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:47:44.738961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:44.738687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:45.107856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.107767 5039 generic.go:356] "Generic (PLEG): container finished" podID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerID="6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a" exitCode=0 Oct 30 21:47:45.108259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.107872 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" Oct 30 21:47:45.108259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.107889 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" event={"ID":"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d","Type":"ContainerDied","Data":"6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a"} Oct 30 21:47:45.108259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.107935 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4" event={"ID":"2c9a100a-efbc-4fe2-a74a-c12ffbfb326d","Type":"ContainerDied","Data":"375dfe126b3ab2292e045a0b4c21dc7a02dce7dbb0916c482b6fb77433c62d38"} Oct 30 21:47:45.108259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.107956 5039 scope.go:122] "RemoveContainer" containerID="6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a" Oct 30 21:47:45.109481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.109452 5039 generic.go:356] "Generic (PLEG): container finished" podID="570a2609-dedf-4228-9fff-3d18d558f715" containerID="2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5" exitCode=0 Oct 30 21:47:45.109602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.109515 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" Oct 30 21:47:45.109602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.109541 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" event={"ID":"570a2609-dedf-4228-9fff-3d18d558f715","Type":"ContainerDied","Data":"2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5"} Oct 30 21:47:45.109602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.109593 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld" event={"ID":"570a2609-dedf-4228-9fff-3d18d558f715","Type":"ContainerDied","Data":"e84976f4c0c559883c2de7304b6290dd796d3c8a517567829bfa87c8b6ff262b"} Oct 30 21:47:45.125153 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.125133 5039 scope.go:122] "RemoveContainer" containerID="6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a" Oct 30 21:47:45.125504 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:45.125480 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a\": container with ID starting with 6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a not found: ID does not exist" containerID="6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a" Oct 30 21:47:45.125567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.125512 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a"} err="failed to get container status \"6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a\": rpc error: code = NotFound desc = could not find container \"6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a\": container with ID starting with 6ebc44362e4ed45da7e9560e703dd5b430c4addfd3dfe1afe108638305006e2a not found: ID does not exist" Oct 30 21:47:45.125567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.125530 5039 scope.go:122] "RemoveContainer" containerID="2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5" Oct 30 21:47:45.136990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.136972 5039 scope.go:122] "RemoveContainer" containerID="2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5" Oct 30 21:47:45.137253 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:45.137233 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5\": container with ID starting with 2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5 not found: ID does not exist" containerID="2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5" Oct 30 21:47:45.137304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.137261 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5"} err="failed to get container status \"2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5\": rpc error: code = NotFound desc = could not find container \"2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5\": container with ID starting with 2ee089c3fadbf20b9625e22a48d21303c48f525ed9124d601492e6a2ccbebbc5 not found: ID does not exist" Oct 30 21:47:45.163796 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.163770 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld"] Oct 30 21:47:45.173442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.173417 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-nv5ld"] Oct 30 21:47:45.212112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.210166 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4"] Oct 30 21:47:45.229176 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:45.229147 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-vsjb4"] Oct 30 21:47:46.815196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:46.815166 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:46.922502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:46.922469 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" path="/var/lib/kubelet/pods/2c9a100a-efbc-4fe2-a74a-c12ffbfb326d/volumes" Oct 30 21:47:46.922967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:46.922946 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570a2609-dedf-4228-9fff-3d18d558f715" path="/var/lib/kubelet/pods/570a2609-dedf-4228-9fff-3d18d558f715/volumes" Oct 30 21:47:47.231583 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:47.231552 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:47.428530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:47.428498 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:48.766084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:48.766039 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:49.019827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:49.019730 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:49.258129 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:49.258096 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:51.069067 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:51.069032 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:52.741484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:52.741453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:54.453158 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:54.453082 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.183985 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:47:55.183948 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:47:55.264382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:55.264349 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:47:56.486331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:56.486270 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:58.229631 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:58.228241 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:58.728454 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:47:58.728422 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:00.083990 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.083949 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:00.196639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.196601 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh"] Oct 30 21:48:00.205988 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.205948 5039 cpu_manager.go:414] "containerMap: removing container" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" Oct 30 21:48:00.206187 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.206171 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" Oct 30 21:48:00.206344 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.206330 5039 cpu_manager.go:414] "containerMap: removing container" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" Oct 30 21:48:00.206483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.206470 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" Oct 30 21:48:00.206828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.206813 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9a100a-efbc-4fe2-a74a-c12ffbfb326d" containerName="istio-proxy" Oct 30 21:48:00.206963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.206951 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="570a2609-dedf-4228-9fff-3d18d558f715" containerName="istio-proxy" Oct 30 21:48:00.228715 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.228685 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.232577 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.232139 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh"] Oct 30 21:48:00.241744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.241719 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-ae6492-openshift-default-dockercfg-xlpqc\"" Oct 30 21:48:00.337749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337671 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337718 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mqrn\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337785 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337817 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337833 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337857 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337880 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.337925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337916 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.338112 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.337955 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439590 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439557 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439599 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439637 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439674 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439714 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439743 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439741 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mqrn\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439815 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439840 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.439916 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.439855 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.440031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.440009 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.440134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.440116 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.440299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.440274 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.440508 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.440447 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.440718 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.440699 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.442191 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.442166 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.444668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.444644 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.477418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.457719 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mqrn\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.477651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.462835 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.546478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.546448 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:00.920655 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:48:00.920624 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6023cae4_55b6_4b8c_baf3_18f30fd21fe2.slice/crio-fb00f04c5543a1762da58c90fd5bde7bcef89997a7b99b40a9b8e6162b8c2c45 WatchSource:0}: Error finding container fb00f04c5543a1762da58c90fd5bde7bcef89997a7b99b40a9b8e6162b8c2c45: Status 404 returned error can't find the container with id fb00f04c5543a1762da58c90fd5bde7bcef89997a7b99b40a9b8e6162b8c2c45 Oct 30 21:48:00.923632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.923588 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:00.923731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.923661 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:00.923731 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.923716 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:00.925080 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:00.925061 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh"] Oct 30 21:48:01.252238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:01.252156 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" event={"ID":"6023cae4-55b6-4b8c-baf3-18f30fd21fe2","Type":"ContainerStarted","Data":"40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2"} Oct 30 21:48:01.252238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:01.252206 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" event={"ID":"6023cae4-55b6-4b8c-baf3-18f30fd21fe2","Type":"ContainerStarted","Data":"fb00f04c5543a1762da58c90fd5bde7bcef89997a7b99b40a9b8e6162b8c2c45"} Oct 30 21:48:01.547541 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:01.547464 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:01.945467 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:01.945426 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:02.549493 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:02.549464 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 21:48:02.549870 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:02.549525 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 21:48:03.521256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:03.521193 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:03.548875 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:03.548824 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 21:48:03.549080 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:03.548894 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 21:48:04.547287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:04.547255 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 21:48:04.547776 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:04.547301 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 21:48:05.329347 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:05.327107 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:05.547170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:05.547127 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 21:48:05.547351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:05.547194 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 21:48:06.307375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.307162 5039 generic.go:356] "Generic (PLEG): container finished" podID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerID="4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb" exitCode=0 Oct 30 21:48:06.307375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.307241 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" event={"ID":"bb3404db-e958-4f7c-b4a2-8edad0eab199","Type":"ContainerDied","Data":"4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb"} Oct 30 21:48:06.308867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.308845 5039 generic.go:356] "Generic (PLEG): container finished" podID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerID="91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617" exitCode=0 Oct 30 21:48:06.308996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.308873 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" event={"ID":"8e574d24-b86a-48cd-af75-0bd5651d0ca2","Type":"ContainerDied","Data":"91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617"} Oct 30 21:48:06.308996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.308886 5039 scope.go:122] "RemoveContainer" containerID="4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb" Oct 30 21:48:06.309238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.309220 5039 scope.go:122] "RemoveContainer" containerID="91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617" Oct 30 21:48:06.311485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.311088 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:06.311485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.311154 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:06.311485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.311186 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:06.311485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.311328 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:06.311485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.311423 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:06.313238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.311469 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:06.393948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.393897 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podStartSLOduration=6.393869875 podStartE2EDuration="6.393869875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:01.317949232 +0000 UTC m=+2320.527888860" watchObservedRunningTime="2026-10-30 21:48:06.393869875 +0000 UTC m=+2325.603809504" Oct 30 21:48:06.547041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.547011 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 21:48:06.547224 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:06.547063 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 21:48:07.201414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.196953 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:07.318828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.318787 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" event={"ID":"8e574d24-b86a-48cd-af75-0bd5651d0ca2","Type":"ContainerStarted","Data":"2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51"} Oct 30 21:48:07.321339 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.321312 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" event={"ID":"bb3404db-e958-4f7c-b4a2-8edad0eab199","Type":"ContainerStarted","Data":"21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064"} Oct 30 21:48:07.551420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.551340 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:07.551850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.551822 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:07.552651 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.552632 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 21:48:07.899428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.897893 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:48:07.899428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.898035 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:48:07.899428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:07.898098 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:48:08.084987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.084957 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:48:08.085320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.085299 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:48:08.085525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.085500 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:48:08.820985 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.820960 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:48:08.846451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.846154 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:08.885774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.885738 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:48:08.886063 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:08.886034 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:48:09.085600 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:09.085501 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:48:09.085915 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:09.085884 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:48:09.882719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:09.882682 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:48:09.883297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:09.883266 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:48:10.083416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:10.082652 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:48:10.083416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:10.082719 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:48:10.692337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:10.692300 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.872424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:10.871070 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.889705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:10.889675 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:48:10.890226 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:10.890199 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:48:11.086551 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:11.086468 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:48:11.086795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:11.086770 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:48:11.903413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:11.885770 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:48:11.903413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:11.885829 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:48:12.082744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.082710 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:48:12.083002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.082976 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:48:12.219513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.219417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.558009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.557930 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.624407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.624361 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb"] Oct 30 21:48:12.630437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.630411 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.646945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.646923 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fdb168-openshift-default-dockercfg-9mvfb\"" Oct 30 21:48:12.663881 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.663847 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb"] Oct 30 21:48:12.711287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.711266 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:48:12.716588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716562 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716604 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716702 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716735 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhnj\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716832 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716919 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716868 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716919 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.716957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.716953 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.717115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.717087 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.745451 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.745420 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.818515 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.818424 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.818829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.818805 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.818975 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.818961 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.819131 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.819108 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.819236 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.819223 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhnj\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.819436 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.819420 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.819607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.819578 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.819822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.819798 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.820044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.820021 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.821227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.820656 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.822292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.821502 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.822292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.821859 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.822292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.822250 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.822634 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.822619 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.822882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.822856 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.824635 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.824608 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.838463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.838438 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhnj\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.845495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.845456 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:12.885682 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.885645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.886571 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.886552 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:48:12.887638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.887622 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 21:48:12.953462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:12.953429 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:13.093599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.093179 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:48:13.288640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.288614 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb"] Oct 30 21:48:13.289824 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:48:13.289798 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3a77f5_d1fa_4b1c_aaaf_80788fad9e3f.slice/crio-65369e8e07c210e73efdd28176eb2e0bb50dacbaf23a99dafed7e8f6454dbefa WatchSource:0}: Error finding container 65369e8e07c210e73efdd28176eb2e0bb50dacbaf23a99dafed7e8f6454dbefa: Status 404 returned error can't find the container with id 65369e8e07c210e73efdd28176eb2e0bb50dacbaf23a99dafed7e8f6454dbefa Oct 30 21:48:13.292598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.292562 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:13.292685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.292631 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:13.292685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.292662 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:13.365762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.365725 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" event={"ID":"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f","Type":"ContainerStarted","Data":"65369e8e07c210e73efdd28176eb2e0bb50dacbaf23a99dafed7e8f6454dbefa"} Oct 30 21:48:13.366970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:13.366951 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 21:48:14.374056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:14.374022 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" event={"ID":"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f","Type":"ContainerStarted","Data":"5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f"} Oct 30 21:48:14.424016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:14.423975 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podStartSLOduration=2.423962385 podStartE2EDuration="2.423962385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.420504899 +0000 UTC m=+2333.630444540" watchObservedRunningTime="2026-10-30 21:48:14.423962385 +0000 UTC m=+2333.633902013" Oct 30 21:48:14.604961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:14.604927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.953901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:14.953873 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:14.959639 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:14.959614 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:15.119021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:15.118965 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:15.379794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:15.379725 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:15.381238 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:15.381216 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:48:15.900923 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:15.900895 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.222719 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.222642 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz"] Oct 30 21:48:16.226730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.226712 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.239589 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.239568 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-846656-serving-cert\"" Oct 30 21:48:16.253430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.253405 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz"] Oct 30 21:48:16.366518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.366488 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbq2\" (UniqueName: \"kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.366685 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.366658 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.366763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.366745 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.366817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.366796 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.370437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.370410 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.468489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.468458 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.469068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.468597 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.469068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.468669 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.469068 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.468753 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbq2\" (UniqueName: \"kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.469599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.469570 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.474192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.474136 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.482544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.482518 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.492833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.492752 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbq2\" (UniqueName: \"kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2\") pod \"maas-api-e2e-extmod-b-846656-796845fcdb-gffpz\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.530058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.530014 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.540946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.540918 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:16.870437 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:16.870404 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz"] Oct 30 21:48:16.870679 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:48:16.870655 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb6bb80_25ca_4984_82af_293b8f254cd3.slice/crio-397736fc4683781224b99707690339a3f4d083f3875cbdf1e1176ed1c3a850fd WatchSource:0}: Error finding container 397736fc4683781224b99707690339a3f4d083f3875cbdf1e1176ed1c3a850fd: Status 404 returned error can't find the container with id 397736fc4683781224b99707690339a3f4d083f3875cbdf1e1176ed1c3a850fd Oct 30 21:48:17.076228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:17.076197 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.317133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:17.317100 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.393094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:17.393060 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" event={"ID":"dcb6bb80-25ca-4984-82af-293b8f254cd3","Type":"ContainerStarted","Data":"39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9"} Oct 30 21:48:17.393287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:17.393115 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" event={"ID":"dcb6bb80-25ca-4984-82af-293b8f254cd3","Type":"ContainerStarted","Data":"397736fc4683781224b99707690339a3f4d083f3875cbdf1e1176ed1c3a850fd"} Oct 30 21:48:17.437411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:17.437350 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" podStartSLOduration=1.1051515570000001 podStartE2EDuration="1.43733651s" totalImagesPullingTime="332.184953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:17.434733865 +0000 UTC m=+2336.644673494" watchObservedRunningTime="2026-10-30 21:48:17.43733651 +0000 UTC m=+2336.647276138" Oct 30 21:48:18.398492 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:18.398462 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:18.476126 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:18.476083 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:18.704698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:18.704587 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:20.118430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:20.118385 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:20.983649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:20.983603 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:21.333320 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:21.333249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:21.549444 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:21.544938 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:22.157431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:22.157373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:22.303231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:22.303195 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:22.447153 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:22.447084 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:22.899012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:22.898969 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:23.200463 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:23.200352 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:23.971647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:23.971611 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:24.410083 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:24.410059 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 21:48:25.184328 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:48:25.184293 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:48:25.521034 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:25.520954 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.813779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:26.813748 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.979607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:26.979578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:27.453861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:27.453805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:28.498710 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:28.498679 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:28.663749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:28.663707 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:29.394359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:29.394324 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:31.087902 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:31.087876 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:32.615550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:32.615514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:32.952095 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:32.952062 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:48:34.544360 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:34.544312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:36.400702 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.400668 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j"] Oct 30 21:48:36.406278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.406259 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.412012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.411967 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:36.422039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.422010 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-846656-openshift-default-dockercfg-sl8ph\"" Oct 30 21:48:36.436669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.436638 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j"] Oct 30 21:48:36.535737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.535708 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.535874 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.535835 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.535925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.535887 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.535925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.535907 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.536031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.535954 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.536128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.536109 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.536164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.536136 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.536222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.536157 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kpc\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.536222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.536210 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.637799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.637777 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.637900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.637827 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.637900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.637844 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.637900 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.637869 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kpc\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.637896 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.637948 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638024 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638085 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638114 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638204 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638313 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638378 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638474 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.638614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.638595 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.640183 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.640159 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.640472 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.640454 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.654525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.654476 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kpc\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.654608 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.654528 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:36.719950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:36.719930 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:37.026735 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.026711 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j"] Oct 30 21:48:37.027385 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:48:37.027359 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a56bfa_062d_4b01_9778_8d9954e1d517.slice/crio-2231a4dcf6a5c1cffcbb8b7992718a0362804e4da6dba5d190035cfb6621961a WatchSource:0}: Error finding container 2231a4dcf6a5c1cffcbb8b7992718a0362804e4da6dba5d190035cfb6621961a: Status 404 returned error can't find the container with id 2231a4dcf6a5c1cffcbb8b7992718a0362804e4da6dba5d190035cfb6621961a Oct 30 21:48:37.029867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.029837 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:37.029928 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.029898 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:37.029965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.029930 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:48:37.530605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.530560 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" event={"ID":"e8a56bfa-062d-4b01-9778-8d9954e1d517","Type":"ContainerStarted","Data":"ef4f55fac712e9c8f6f948e77588c9671d6c7348940a9a37a49c7433f0b814af"} Oct 30 21:48:37.531116 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.530623 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" event={"ID":"e8a56bfa-062d-4b01-9778-8d9954e1d517","Type":"ContainerStarted","Data":"2231a4dcf6a5c1cffcbb8b7992718a0362804e4da6dba5d190035cfb6621961a"} Oct 30 21:48:37.583414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.583186 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" podStartSLOduration=1.583167196 podStartE2EDuration="1.583167196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:37.577713101 +0000 UTC m=+2356.787652730" watchObservedRunningTime="2026-10-30 21:48:37.583167196 +0000 UTC m=+2356.793106825" Oct 30 21:48:37.720727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:37.720693 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:38.080002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:38.079919 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:38.721769 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:38.721736 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:48:38.722230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:38.721792 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:48:39.720449 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:39.720411 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:48:39.720625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:39.720481 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:48:39.828373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:39.828343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:40.720673 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:40.720643 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:48:40.720840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:40.720696 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:48:41.592217 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:41.592181 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:41.720623 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:41.720590 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:48:41.720805 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:41.720652 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:48:42.774832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:42.774800 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:42.775324 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:42.775288 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:42.776512 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:42.776487 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 21:48:43.431963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:43.431927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:44.145527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:44.145491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:45.467796 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:45.467760 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:47.544380 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:47.544338 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:49.540688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:49.540648 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.307612 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:51.307547 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.812059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.812024 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh"] Oct 30 21:48:52.816497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.816474 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:52.841425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.837743 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fdb168-serving-cert\"" Oct 30 21:48:52.846739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.846713 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh"] Oct 30 21:48:52.935753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.935705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.964497 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.964464 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbghn\" (UniqueName: \"kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:52.964661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.964643 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:52.964730 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.964682 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:52.964795 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:52.964768 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.065699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.065619 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbghn\" (UniqueName: \"kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.065863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.065731 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.065863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.065771 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.065863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.065800 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.066379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.066352 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.068155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.068133 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.068277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.068179 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.082547 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.082527 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbghn\" (UniqueName: \"kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn\") pod \"maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.089699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.089677 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:53.130524 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.130503 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:53.239616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.239582 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:53.426619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.426591 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh"] Oct 30 21:48:53.427037 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:48:53.427012 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1bb836_9911_4a78_9534_214c4916bf49.slice/crio-1a554b198ab93f429c855df6346634448c01be78ca892cea7b1f7c4ad8dc759d WatchSource:0}: Error finding container 1a554b198ab93f429c855df6346634448c01be78ca892cea7b1f7c4ad8dc759d: Status 404 returned error can't find the container with id 1a554b198ab93f429c855df6346634448c01be78ca892cea7b1f7c4ad8dc759d Oct 30 21:48:53.655371 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:53.655328 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" event={"ID":"fa1bb836-9911-4a78-9534-214c4916bf49","Type":"ContainerStarted","Data":"1a554b198ab93f429c855df6346634448c01be78ca892cea7b1f7c4ad8dc759d"} Oct 30 21:48:54.662271 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:54.662238 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" event={"ID":"fa1bb836-9911-4a78-9534-214c4916bf49","Type":"ContainerStarted","Data":"6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c"} Oct 30 21:48:54.662732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:54.662439 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:48:54.696817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:54.696771 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" podStartSLOduration=2.39445146 podStartE2EDuration="2.696757684s" totalImagesPullingTime="302.306224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:54.696122072 +0000 UTC m=+2373.906061700" watchObservedRunningTime="2026-10-30 21:48:54.696757684 +0000 UTC m=+2373.906697314" Oct 30 21:48:54.728134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:54.728104 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:55.182162 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:48:55.182126 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:48:55.515838 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:55.515751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:55.671977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:55.671944 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:56.634794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:56.634756 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:57.774770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:57.774736 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:58.370944 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:58.370907 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:58.970658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:58.970626 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.128143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:59.128110 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.279509 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:59.279414 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.544546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:59.544472 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.703214 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:59.703178 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:59.860568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:48:59.860481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.000854 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:00.000828 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.350653 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:00.350617 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.492982 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:00.492945 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:00.676363 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:00.676336 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 21:49:01.898647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:01.898608 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:02.291603 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:02.291522 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.875861 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:03.875790 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.265169 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:04.265133 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.415691 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:04.415653 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.643416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:04.643359 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.211909 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:05.211868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.677168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:05.677135 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:05.842410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:05.841126 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:06.259750 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:06.259718 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:49:06.307808 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:06.307771 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.105977 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:08.105924 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:09.717991 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:09.717958 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:09.938132 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:09.938096 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:10.097352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:10.097266 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:11.794997 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:11.794899 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:11.949210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:11.949172 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:13.408025 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:13.407992 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:14.745298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:14.745257 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n"] Oct 30 21:49:14.745845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:14.745544 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" containerID="cri-o://10fe92eb3e4aa8dddc73531631a830863433936289f3fa7f37d9a780494bcf48" gracePeriod=30 Oct 30 21:49:15.003352 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:15.003257 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:15.219913 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:15.219878 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:16.421509 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.421477 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:49:16.422027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.421532 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:16.509431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.509371 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:16.838430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.838349 5039 generic.go:356] "Generic (PLEG): container finished" podID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerID="5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f" exitCode=0 Oct 30 21:49:16.838430 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.838386 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" event={"ID":"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f","Type":"ContainerDied","Data":"5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f"} Oct 30 21:49:16.838779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.838764 5039 scope.go:122] "RemoveContainer" containerID="5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f" Oct 30 21:49:16.840336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.840308 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:16.840423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.840382 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:16.840464 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:16.840447 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:17.847289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:17.847245 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" event={"ID":"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f","Type":"ContainerStarted","Data":"49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8"} Oct 30 21:49:17.954147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:17.954114 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:49:17.954333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:17.954215 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:49:17.954333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:17.954263 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:49:18.236607 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:18.236570 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:18.954102 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:18.954067 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:49:18.954517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:18.954117 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:49:19.888181 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:19.888155 5039 generic.go:356] "Generic (PLEG): container finished" podID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerID="10fe92eb3e4aa8dddc73531631a830863433936289f3fa7f37d9a780494bcf48" exitCode=0 Oct 30 21:49:19.888319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:19.888231 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" event={"ID":"bfdabfed-f775-408b-9ab9-b9e5cc1e653f","Type":"ContainerDied","Data":"10fe92eb3e4aa8dddc73531631a830863433936289f3fa7f37d9a780494bcf48"} Oct 30 21:49:19.893591 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:19.893563 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:19.955748 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:19.955711 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:49:19.956159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:19.955786 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:49:20.091113 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.091086 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:49:20.126797 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126770 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.126915 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126877 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.126915 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126896 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126927 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126949 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127001 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126975 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.126998 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:20.127147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127033 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127147 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127116 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127198 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg\") pod \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\" (UID: \"bfdabfed-f775-408b-9ab9-b9e5cc1e653f\") " Oct 30 21:49:20.127301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127209 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:20.127301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127269 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:20.127461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127338 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:20.127593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127569 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:20.127779 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127761 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.127862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127782 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.127862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127791 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.127862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127805 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.127862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.127817 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.129142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.129116 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:20.129326 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.129305 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "kube-api-access-68bhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:20.129470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.129451 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:20.129549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.129463 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token" pod "bfdabfed-f775-408b-9ab9-b9e5cc1e653f" (UID: "bfdabfed-f775-408b-9ab9-b9e5cc1e653f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:20.229294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.229265 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.229294 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.229292 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.229513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.229302 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.229513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.229311 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68bhg\" (UniqueName: \"kubernetes.io/projected/bfdabfed-f775-408b-9ab9-b9e5cc1e653f-kube-api-access-68bhg\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.897656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.897621 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" event={"ID":"bfdabfed-f775-408b-9ab9-b9e5cc1e653f","Type":"ContainerDied","Data":"3abd468c37043a516beaaeca697d105bd797dd04f3921ee42ffa993f2a6b017c"} Oct 30 21:49:20.897656 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.897641 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n" Oct 30 21:49:20.897968 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.897685 5039 scope.go:122] "RemoveContainer" containerID="10fe92eb3e4aa8dddc73531631a830863433936289f3fa7f37d9a780494bcf48" Oct 30 21:49:20.951869 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:49:20.951834 5039 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdabfed_f775_408b_9ab9_b9e5cc1e653f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdabfed_f775_408b_9ab9_b9e5cc1e653f.slice/crio-3abd468c37043a516beaaeca697d105bd797dd04f3921ee42ffa993f2a6b017c\": RecentStats: unable to find data in memory cache]" Oct 30 21:49:20.953672 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.953649 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:49:20.953770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.953688 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:49:20.953770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.953711 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n"] Oct 30 21:49:20.986078 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:20.986040 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-7r97n"] Oct 30 21:49:21.286077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:21.285989 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:49:21.300383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:21.300358 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:49:21.393499 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:21.393468 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:21.954143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:21.954113 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:49:21.954349 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:21.954171 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:49:22.912087 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:22.912054 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:22.936948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:22.936866 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" path="/var/lib/kubelet/pods/bfdabfed-f775-408b-9ab9-b9e5cc1e653f/volumes" Oct 30 21:49:22.958679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:22.958654 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:49:22.959069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:22.959045 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:49:22.959785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:22.959765 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 21:49:24.131777 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.131735 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v"] Oct 30 21:49:24.132256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.131949 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" podUID="f1b0633b-746d-4660-ac74-5582983f60f4" containerName="maas-api" containerID="cri-o://21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06" gracePeriod=30 Oct 30 21:49:24.242541 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.242500 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:24.442205 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.442182 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:49:24.587931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.587891 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls\") pod \"f1b0633b-746d-4660-ac74-5582983f60f4\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " Oct 30 21:49:24.588108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.588025 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca\") pod \"f1b0633b-746d-4660-ac74-5582983f60f4\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " Oct 30 21:49:24.588108 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.588079 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6\" (UniqueName: \"kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6\") pod \"f1b0633b-746d-4660-ac74-5582983f60f4\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " Oct 30 21:49:24.588429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.588380 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs\") pod \"f1b0633b-746d-4660-ac74-5582983f60f4\" (UID: \"f1b0633b-746d-4660-ac74-5582983f60f4\") " Oct 30 21:49:24.591546 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.591488 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs" pod "f1b0633b-746d-4660-ac74-5582983f60f4" (UID: "f1b0633b-746d-4660-ac74-5582983f60f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:24.608764 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.593154 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca" pod "f1b0633b-746d-4660-ac74-5582983f60f4" (UID: "f1b0633b-746d-4660-ac74-5582983f60f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:24.610057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.610026 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls" pod "f1b0633b-746d-4660-ac74-5582983f60f4" (UID: "f1b0633b-746d-4660-ac74-5582983f60f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:24.625569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.624680 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6" pod "f1b0633b-746d-4660-ac74-5582983f60f4" (UID: "f1b0633b-746d-4660-ac74-5582983f60f4"). InnerVolumeSpecName "kube-api-access-w4ct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:24.690774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.690702 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:24.690774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.690727 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1b0633b-746d-4660-ac74-5582983f60f4-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:24.690774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.690739 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b0633b-746d-4660-ac74-5582983f60f4-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:24.690774 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.690751 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ct6\" (UniqueName: \"kubernetes.io/projected/f1b0633b-746d-4660-ac74-5582983f60f4-kube-api-access-w4ct6\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 21:49:24.854913 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.854879 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:24.925262 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.925233 5039 generic.go:356] "Generic (PLEG): container finished" podID="f1b0633b-746d-4660-ac74-5582983f60f4" containerID="21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06" exitCode=0 Oct 30 21:49:24.925422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.925295 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" Oct 30 21:49:24.925422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.925311 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" event={"ID":"f1b0633b-746d-4660-ac74-5582983f60f4","Type":"ContainerDied","Data":"21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06"} Oct 30 21:49:24.925422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.925356 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v" event={"ID":"f1b0633b-746d-4660-ac74-5582983f60f4","Type":"ContainerDied","Data":"da2f09085f30cfb4ed545dc7d139a828a5f8e29ff12c1a65bca8cd68271e7ace"} Oct 30 21:49:24.925422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.925383 5039 scope.go:122] "RemoveContainer" containerID="21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06" Oct 30 21:49:24.938300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.938277 5039 scope.go:122] "RemoveContainer" containerID="21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06" Oct 30 21:49:24.938795 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:49:24.938767 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06\": container with ID starting with 21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06 not found: ID does not exist" containerID="21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06" Oct 30 21:49:24.938873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.938801 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06"} err="failed to get container status \"21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06\": rpc error: code = NotFound desc = could not find container \"21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06\": container with ID starting with 21afaf8155695a456c7edb6d993c6c021159b913f04217476c670e74be7f0d06 not found: ID does not exist" Oct 30 21:49:24.971469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.971416 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v"] Oct 30 21:49:24.975103 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:24.975083 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3bc8ea-5c694d96fb-zr66v"] Oct 30 21:49:25.005567 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:25.005534 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:25.185279 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:49:25.185241 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:49:25.448717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:25.448687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:26.610624 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:26.610586 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:26.922941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:26.922912 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b0633b-746d-4660-ac74-5582983f60f4" path="/var/lib/kubelet/pods/f1b0633b-746d-4660-ac74-5582983f60f4/volumes" Oct 30 21:49:28.207414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:28.207354 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.983753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:29.983713 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.561469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:31.561432 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.726209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:31.726165 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.880383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:31.880302 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.486255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:33.486214 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.655128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:33.655090 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.795780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:33.795705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.117530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.117446 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l"] Oct 30 21:49:34.118276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.118247 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" Oct 30 21:49:34.118276 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.118275 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" Oct 30 21:49:34.118496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.118410 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f1b0633b-746d-4660-ac74-5582983f60f4" containerName="maas-api" Oct 30 21:49:34.118496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.118423 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b0633b-746d-4660-ac74-5582983f60f4" containerName="maas-api" Oct 30 21:49:34.118606 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.118579 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b0633b-746d-4660-ac74-5582983f60f4" containerName="maas-api" Oct 30 21:49:34.118606 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.118595 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdabfed-f775-408b-9ab9-b9e5cc1e653f" containerName="istio-proxy" Oct 30 21:49:34.122552 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.122529 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.136032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.135995 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5d3601-openshift-default-dockercfg-4s794\"" Oct 30 21:49:34.150914 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.150892 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l"] Oct 30 21:49:34.199692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199659 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.199827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199712 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.199827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199786 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.199827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199818 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.199948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199845 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.199948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199878 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6pmm\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.199948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199936 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.200039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.199985 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.200039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.200028 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.223375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.223341 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.301526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301503 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301569 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301640 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301596 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301645 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301669 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301694 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301731 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6pmm\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301790 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.301918 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.301855 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.302367 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.302340 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.302510 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.302414 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.302650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.302629 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.302725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.302703 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.302780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.302728 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.304230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.304211 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.304629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.304615 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.352426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.347996 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.352426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.349126 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6pmm\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.375468 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.374688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.439785 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.439605 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:34.767258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.767225 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l"] Oct 30 21:49:34.768243 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:49:34.768214 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b968a90_d6dd_481f_b845_8bc0463ffae5.slice/crio-f79197ada53f1cc70f740db13db4f827848c1f5b4a128b4370fa7a686525547b WatchSource:0}: Error finding container f79197ada53f1cc70f740db13db4f827848c1f5b4a128b4370fa7a686525547b: Status 404 returned error can't find the container with id f79197ada53f1cc70f740db13db4f827848c1f5b4a128b4370fa7a686525547b Oct 30 21:49:34.771661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.771624 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.771749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.771693 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.771749 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.771723 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.869006 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.868974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.993885 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.993846 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" event={"ID":"3b968a90-d6dd-481f-b845-8bc0463ffae5","Type":"ContainerStarted","Data":"d3edc710d828e9b2e69656ca273591b91a18687a412d23ec76bcf27bd0512647"} Oct 30 21:49:34.994044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:34.993925 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" event={"ID":"3b968a90-d6dd-481f-b845-8bc0463ffae5","Type":"ContainerStarted","Data":"f79197ada53f1cc70f740db13db4f827848c1f5b4a128b4370fa7a686525547b"} Oct 30 21:49:35.021009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.020939 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.034619 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.034566 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podStartSLOduration=1.034549557 podStartE2EDuration="1.034549557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.032125614 +0000 UTC m=+2414.242065246" watchObservedRunningTime="2026-10-30 21:49:35.034549557 +0000 UTC m=+2414.244489186" Oct 30 21:49:35.173783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.173749 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.348568 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.348482 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.440637 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.440608 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:35.446328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.446303 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:35.564244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.564210 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.711859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.711831 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.999692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:35.999628 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:36.000575 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:36.000554 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:49:36.177377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:36.177342 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:49:36.307482 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:36.301004 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:36.447893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:36.447863 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.822542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:37.822506 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.005695 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:38.005662 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.938814 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:39.938776 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.123628 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:40.123584 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.719528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:41.719489 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.491527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:43.491467 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.152208 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:45.152171 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.539296 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.539266 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp"] Oct 30 21:49:46.544502 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.544480 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.563202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.563169 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-9722ac-serving-cert\"" Oct 30 21:49:46.576228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.576198 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp"] Oct 30 21:49:46.674957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.674925 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.709883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.709851 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.710009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.709929 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.710064 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.710012 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.710114 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.710096 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsnv\" (UniqueName: \"kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.811297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.811219 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.811297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.811261 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.811561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.811301 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.811561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.811340 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsnv\" (UniqueName: \"kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.811893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.811870 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.813850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.813830 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.813973 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.813952 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.817815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.817748 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.827617 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.827595 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsnv\" (UniqueName: \"kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv\") pod \"maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:46.858614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:46.858593 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:47.155613 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:47.155586 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp"] Oct 30 21:49:47.156538 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:49:47.156518 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfddd2d5_b5c9_46ff_915f_7ce09e2ab5d6.slice/crio-1ec3cd7195485a697a0ed962e19cd526a8df4c613dfe3a7dbbef135007ff6db6 WatchSource:0}: Error finding container 1ec3cd7195485a697a0ed962e19cd526a8df4c613dfe3a7dbbef135007ff6db6: Status 404 returned error can't find the container with id 1ec3cd7195485a697a0ed962e19cd526a8df4c613dfe3a7dbbef135007ff6db6 Oct 30 21:49:48.072905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:48.072824 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" event={"ID":"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6","Type":"ContainerStarted","Data":"b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9"} Oct 30 21:49:48.072905 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:48.072883 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" event={"ID":"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6","Type":"ContainerStarted","Data":"1ec3cd7195485a697a0ed962e19cd526a8df4c613dfe3a7dbbef135007ff6db6"} Oct 30 21:49:48.073350 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:48.073065 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:48.110699 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:48.110649 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" podStartSLOduration=1.6148549189999999 podStartE2EDuration="2.110635709s" totalImagesPullingTime="495.78079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:48.107167944 +0000 UTC m=+2427.317107575" watchObservedRunningTime="2026-10-30 21:49:48.110635709 +0000 UTC m=+2427.320575337" Oct 30 21:49:48.304886 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:48.304853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.650422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:48.650355 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:49.868354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:49.868310 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.085592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:50.085562 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.245597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:50.245558 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.499179 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:50.499087 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.654873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:50.654838 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.814100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:50.814006 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.340172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:51.340129 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.487981 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:51.487948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.909031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:51.908997 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.803601 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:52.803551 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:53.612084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:53.612049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.083659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:54.083635 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 21:49:55.185096 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:49:55.185061 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:49:55.274257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:55.274229 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.886434 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:56.886376 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:58.455447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:49:58.455410 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.092722 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:00.092684 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:01.610650 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:01.610617 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.185489 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:03.185457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.793681 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:04.793646 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.400364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:06.400333 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.767811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:06.767710 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:50:08.248484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:08.248442 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.888478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:09.888432 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.531933 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:11.531900 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.270847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:13.270808 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:14.966364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:14.966334 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.105002 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:15.104968 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.246368 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:15.246284 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.382089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:15.382047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:16.729920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:16.729859 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:18.285614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:18.285582 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:19.879999 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:19.879961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:21.624955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:21.624919 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:23.314661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:23.314613 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:24.951407 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:24.951368 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:25.184480 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:50:25.184443 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:50:26.713646 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:26.713609 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.179016 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:28.178987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.385245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:28.385201 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.553060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:28.552961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.726458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:28.726426 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.287581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:29.287540 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.438780 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:29.438746 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.194520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:30.194476 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.333237 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:30.333202 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:31.511931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:31.511816 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:32.006140 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:32.006100 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:32.704144 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:32.704075 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:32.874239 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:32.874213 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.128500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:33.128377 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.322373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:33.322337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.484462 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:33.484431 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.830038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:33.829934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.127843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:34.127750 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.287878 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:34.287838 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.489648 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:35.489616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.638408 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:35.638362 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:37.286522 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:37.286490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.904190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:38.904144 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:40.460713 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:40.460674 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:41.963024 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:41.962989 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:43.611887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:43.611853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.199335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:45.199299 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:46.689259 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:46.689219 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:46.824491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:46.824454 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:48.283922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:48.283892 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:49.904353 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:49.904312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.411935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:51.411902 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:53.009135 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:53.009089 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:54.355287 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:54.355250 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:50:54.726150 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:54.726115 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.184970 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:50:55.184886 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:50:56.156715 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:56.156683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:56.298053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:56.298018 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:56.655644 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:56.655610 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.190433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:58.190212 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.334490 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:59.334453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.485301 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:59.485169 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.640833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:59.640800 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.851703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:50:59.851661 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.022832 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:00.022742 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.197586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:00.197546 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.354807 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:00.354697 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.701128 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:00.701078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.254190 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:01.254158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.745319 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:01.745281 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.062839 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:02.062768 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.284812 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:02.284779 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.430930 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:02.430887 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.079138 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:03.079098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.226576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:03.226537 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.980408 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:03.980355 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.517305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.517274 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.704351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.704313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.717668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.717640 5039 generic.go:356] "Generic (PLEG): container finished" podID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerID="d3edc710d828e9b2e69656ca273591b91a18687a412d23ec76bcf27bd0512647" exitCode=0 Oct 30 21:51:05.717960 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.717709 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" event={"ID":"3b968a90-d6dd-481f-b845-8bc0463ffae5","Type":"ContainerDied","Data":"d3edc710d828e9b2e69656ca273591b91a18687a412d23ec76bcf27bd0512647"} Oct 30 21:51:05.718531 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.718511 5039 scope.go:122] "RemoveContainer" containerID="d3edc710d828e9b2e69656ca273591b91a18687a412d23ec76bcf27bd0512647" Oct 30 21:51:05.721517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.721480 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:05.721649 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.721571 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:05.721720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.721707 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:05.999668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:05.999640 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:51:06.727284 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:06.727239 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" event={"ID":"3b968a90-d6dd-481f-b845-8bc0463ffae5","Type":"ContainerStarted","Data":"11ff86d0a16aa8cb0ece0ca34603ae3574baeb85f18b694b46408adc2dd3de65"} Oct 30 21:51:07.283343 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:07.283304 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:07.440869 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:07.440836 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:51:07.441723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:07.441695 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:51:07.441847 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:07.441755 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 21:51:07.444004 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:07.443975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:07.655257 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:07.655206 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.440995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:08.440961 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:51:08.441373 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:08.441008 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 21:51:09.195592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:09.195558 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:09.441461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:09.441429 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:51:09.441833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:09.441474 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 21:51:10.440705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:10.440675 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:51:10.440877 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:10.440722 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 21:51:10.790310 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:10.790185 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:11.441362 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:11.441334 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:51:11.441618 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:11.441381 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 21:51:12.296436 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:12.296370 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:12.446047 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:12.446019 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:51:12.446445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:12.446417 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:51:12.447084 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:12.447065 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 21:51:12.897170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:12.897131 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:13.922755 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:13.922716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.481993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:15.481959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.645595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:15.645497 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:17.391152 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:17.391114 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.090759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:19.090721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:20.682300 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:20.682267 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:22.300057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:22.300009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:23.960356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:23.960324 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:25.183979 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:51:25.183942 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:51:25.590140 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:25.590059 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:27.233868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:27.233803 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:28.831871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:28.831835 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:30.453827 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:30.453789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:32.078247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:32.078203 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:33.682033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:33.681995 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.631063 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:34.631025 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.280374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:35.280336 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:36.874880 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:36.874844 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.492633 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:38.492598 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.097323 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:40.097286 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.731548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:41.731512 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.162958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.162925 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:51:43.322457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.322432 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk"] Oct 30 21:51:43.327675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.327654 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.344599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.344572 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ed0866-serving-cert\"" Oct 30 21:51:43.348850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.348829 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk"] Oct 30 21:51:43.386543 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.386506 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.386705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.386681 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gp4g\" (UniqueName: \"kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.386862 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.386843 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.386926 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.386907 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.468270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.468194 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.487789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.487768 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.487904 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.487806 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.488030 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.487943 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gp4g\" (UniqueName: \"kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.488133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.488047 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.488208 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:51:43.488190 5039 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-serving-cert: secret "maas-api-e2e-extmod-b-ed0866-serving-cert" not found Oct 30 21:51:43.488272 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:51:43.488262 5039 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls podName:aa6ff71c-06fb-4546-a65e-d0ac053f8637 nodeName:}" failed. No retries permitted until 2026-10-30 21:51:43.98823985 +0000 UTC m=+2543.198179470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls") pod "maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" (UID: "aa6ff71c-06fb-4546-a65e-d0ac053f8637") : secret "maas-api-e2e-extmod-b-ed0866-serving-cert" not found Oct 30 21:51:43.488605 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.488588 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.490192 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.490176 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.513557 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.509496 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gp4g\" (UniqueName: \"kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.631545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.631507 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.789071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.788993 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.996483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.996441 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:43.998836 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:43.998813 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:44.241182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:44.241148 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:44.572516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:44.572415 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk"] Oct 30 21:51:44.986632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:44.986601 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" event={"ID":"aa6ff71c-06fb-4546-a65e-d0ac053f8637","Type":"ContainerStarted","Data":"c5b997edad21c5b50a86a7791394c810f2c565475608c7d94e24b11c0072dd55"} Oct 30 21:51:45.294473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:45.294026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.961426 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:45.953224 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.996561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:45.996523 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" event={"ID":"aa6ff71c-06fb-4546-a65e-d0ac053f8637","Type":"ContainerStarted","Data":"3f82f170f4d14ded5ca998ded9700b01474d0844a4a950914fe12d1946bb1c78"} Oct 30 21:51:45.996759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:45.996728 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:46.037149 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:46.037100 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" podStartSLOduration=2.628403414 podStartE2EDuration="3.03708723s" totalImagesPullingTime="408.683816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:46.036107354 +0000 UTC m=+2545.246046983" watchObservedRunningTime="2026-10-30 21:51:46.03708723 +0000 UTC m=+2545.247026868" Oct 30 21:51:47.106209 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:47.106172 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.796071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:47.796027 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.934598 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:47.934568 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.140470 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:48.136107 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.900642 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:48.900607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:49.329210 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:49.329049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.842182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:50.842153 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.983887 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:50.983852 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:51.173561 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:51.173527 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:52.007125 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:52.007097 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 21:51:52.833039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:52.833001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:53.209309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:53.209272 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:53.353483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:53.353444 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:54.152668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:54.152633 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:54.877513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:54.877415 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:55.184377 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:51:55.184345 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:51:55.584969 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:55.584899 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:55.721843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:55.721813 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:56.347148 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:56.347117 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:56.501604 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:56.501574 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.083525 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:58.083490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.228224 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:58.228183 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.693970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:51:59.693934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.348247 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:00.348203 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.505534 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:00.505494 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.658162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:00.658114 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:01.642385 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:01.642351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:03.353666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:03.353585 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:05.017415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:05.017368 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:05.707007 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:05.706975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:05.876355 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:05.876320 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.322069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:07.322034 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:08.384386 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:08.384344 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:09.197845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:09.197792 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:10.880445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:10.880411 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:12.401666 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:12.401621 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:12.553408 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:12.553366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:12.697440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:12.697369 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:13.088298 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:13.088217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:14.407614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:14.407564 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:15.083630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:15.083597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:15.255200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:15.255159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:15.606962 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:15.606929 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:16.222929 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:16.222889 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:17.955787 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:17.955708 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:19.670458 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:19.670418 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:20.310945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:20.310912 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:20.539369 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:20.539335 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:21.462185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:21.462147 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:23.147950 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:23.147911 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:24.893137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:24.893081 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:25.185407 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:52:25.185304 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:52:26.523678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:26.523636 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:27.481231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:27.481184 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:27.682375 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:27.682347 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:27.858420 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:27.858321 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.051602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:28.051550 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.243056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:28.243018 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.465172 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:28.465138 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.774817 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:28.774782 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:29.147922 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:29.147880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:29.336137 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:29.335198 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:30.526165 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:30.526127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:30.807457 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:30.807335 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:31.721295 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:31.721260 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:31.897411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:31.897363 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:32.117949 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:32.117868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:32.401778 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:32.401751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:32.670455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:32.670353 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:32.859242 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:32.859177 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:33.584090 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:33.584054 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:33.808168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:33.808136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:34.083670 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:34.083636 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:34.372336 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:34.372252 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:34.539811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:34.539777 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:34.737082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:34.737045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:34.958758 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:34.958723 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:35.111250 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:35.111156 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:36.682518 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:36.682479 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:37.329131 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:37.329096 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:38.467055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:38.467023 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:40.112009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:40.111970 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:41.768104 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:41.768064 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:43.242698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:43.242664 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:52:43.425085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:43.425047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:45.290171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:45.289983 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:46.300089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:46.297721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:47.189406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:47.189371 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:47.374091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:47.374045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:47.852370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:47.852340 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:48.327698 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:48.327662 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:48.601566 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:48.601477 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:49.367967 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:49.367932 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.344578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:50.344538 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.498530 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:50.498344 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.666599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:50.666562 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.997491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:50.997409 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:51.463616 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:51.463588 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:52.177198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:52.177163 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:53.124230 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:53.124194 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:54.790593 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:54.790560 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:55.184254 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:52:55.184218 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:52:56.367167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:56.367128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:57.454057 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:57.454025 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:57.603379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:57.603345 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:57.797945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:57.797853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:58.114753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:58.114670 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:58.913668 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:58.913634 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:59.902739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:52:59.902704 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:01.420168 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:01.420132 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:03.063074 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:03.063037 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:03.609325 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:03.609285 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:03.771725 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:03.771694 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:04.094533 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:04.094510 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:04.346596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:04.346512 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.027686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:05.027657 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.181819 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:05.181783 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.518476 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:05.518440 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.807485 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:05.807408 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:06.894986 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:06.894951 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:08.500422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:08.500353 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:10.136661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:10.136604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:10.905580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:10.905542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:11.095641 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:11.095606 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:12.101537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:12.101496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:13.508283 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:13.508252 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:13.957810 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:13.957761 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:15.668500 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:15.668462 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:15.841055 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:15.841022 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:17.411751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:17.411713 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:19.344602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:19.344555 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:20.651972 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:20.651934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:20.840734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:20.840687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:21.440580 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:21.440545 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:23.090744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:23.085417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.675299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:24.675244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:25.184245 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:53:25.184206 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:53:26.324185 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:26.324150 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:27.919023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:27.918982 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:28.614202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:28.614160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:28.769821 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:28.769789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:29.454688 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:29.454630 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:31.133265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:31.133227 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:32.785448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:32.785231 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:34.314680 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:34.314642 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:35.927442 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:35.927406 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:37.490267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:37.490226 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:39.137023 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:39.136986 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:40.698935 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:40.698756 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:42.238496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:42.238456 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:43.755218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:43.755179 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:44.297599 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:44.297559 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:44.434086 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:44.434042 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:45.497564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:45.497530 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:47.003860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:47.003822 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:48.364031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:48.363999 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:53:48.644448 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:48.644407 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.309435 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:50.309342 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.898516 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:51.898481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:53.473652 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:53.473616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.102869 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:55.102836 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.184067 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:53:55.184030 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:53:56.204946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:56.204792 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:56.347275 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:56.347240 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:56.866267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:56.866235 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:57.092150 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:57.092120 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:58.788705 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:53:58.788666 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:00.323883 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:00.323846 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.041893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:02.041840 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.091445 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:03.091406 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.262739 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:03.262705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.537184 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:03.537158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.700159 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:03.700112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:05.297520 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:05.297487 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.827898 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:06.827862 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:08.445772 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:08.445739 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:08.971199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:08.971159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:09.111289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:09.111247 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:10.158592 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:10.158558 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:10.325077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:10.325044 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:10.509868 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:10.509794 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:10.684663 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:10.684624 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:10.867948 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:10.867861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.369309 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:11.369273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.908089 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:11.908054 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:12.515542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:12.515503 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:14.213044 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:14.213008 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:15.802945 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:15.802906 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:15.954661 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:15.954628 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.489307 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:17.489268 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.672738 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:17.672701 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:19.293716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:19.293678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:20.915333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:20.915293 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:21.330842 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:21.330762 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:54:21.344789 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:21.344767 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:54:22.497848 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:22.497810 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:24.222425 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:24.222357 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:25.183852 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:54:25.183819 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:54:25.757882 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:25.757845 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:27.420049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:27.420013 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:29.160365 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:29.160322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:30.804415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:30.804367 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:32.488602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:32.488558 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:34.027297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:34.027249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:35.702775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:35.702737 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:36.702840 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:36.702807 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:54:37.339418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:37.339364 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:39.057528 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:39.057486 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.423202 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:40.423162 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.575757 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:40.575721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.876974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:40.876858 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:41.707608 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:41.707563 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:41.857784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:41.857747 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:42.677545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:42.677501 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:42.862289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:42.862248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:43.011142 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:43.011034 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:43.256865 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:43.256830 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:44.951115 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:44.951056 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:46.508374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:46.507076 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:48.069370 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:48.069328 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:48.221243 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:48.221192 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:48.413354 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:48.413315 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:49.008692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:49.008659 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:50.114697 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:50.114653 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:50.292438 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:50.292385 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:51.051013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:51.050979 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:51.216161 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:51.216125 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:52.086233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:52.086190 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:52.399677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:52.399645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:52.567270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:52.567230 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:53.172304 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:53.172262 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:53.467908 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:53.467826 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:53.610669 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:53.610633 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:53.874625 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:53.874553 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.048378 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:54.048336 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.270447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:54.270415 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.467022 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:54.466990 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.679744 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:54.679711 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:55.160013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:55.159971 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:55.187239 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:54:55.187201 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:54:55.327323 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:55.327231 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:56.205995 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:56.205964 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:56.383790 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:56.383740 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:56.555833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:56.555745 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:56.747413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:56.744897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:58.510614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:54:58.510548 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:00.088533 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:00.088497 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:01.600223 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:01.600192 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:03.213741 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:03.213319 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.009549 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:04.009517 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.196720 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:04.196683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.344245 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:04.344159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.609942 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:04.609859 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.914538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:04.914511 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:05.122453 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:05.122416 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:05.264523 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:05.264447 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:06.693517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:06.693482 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:08.338251 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:08.338223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:10.028027 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:10.027996 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:10.980205 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:10.980167 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.138612 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:11.138575 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.283478 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:11.283366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.436519 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:11.436488 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.678465 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:11.678433 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:13.291569 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:13.291528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:14.991201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:14.991160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:15.944351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:15.944313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:16.577194 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:16.577145 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:17.261134 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:17.261092 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:17.483481 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:17.483444 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:18.168036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:18.168001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:18.377937 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:18.377894 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:18.599871 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:18.599779 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:18.864150 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:18.864078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:19.106629 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:19.106591 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:20.400891 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:20.400840 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:20.603974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:20.603926 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.078331 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:21.078294 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.549446 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:21.549405 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.765036 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:21.764993 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:22.149884 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:22.149848 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:23.693326 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:23.693276 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:25.184634 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:55:25.184596 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:55:25.427527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:25.427490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:26.685925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:26.685891 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:55:27.167085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:27.167047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:29.421227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:29.421152 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:31.173675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:31.173643 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:31.590235 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:31.590141 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:31.796049 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:31.795739 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:32.964255 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:32.964217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:34.829763 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:34.829725 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:36.645455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:36.645414 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:38.506166 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:38.506123 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:40.147709 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:40.147658 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:41.797452 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:41.797419 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:43.705917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:43.705880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:45.672019 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:45.671984 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:47.931397 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:47.931345 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:50.048447 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:50.048408 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:52.018595 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:52.018553 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:53.736857 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:53.736803 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:54.821093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:54.821051 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.079379 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:55.079285 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.185955 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:55:55.185920 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:55:55.243662 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:55.243623 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.525867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:55.525835 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.683679 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:55.683650 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.911339 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:55.911304 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.131548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:56.131514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.302270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:56.302189 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.940974 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:56.940938 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:57.258409 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:57.258299 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:57.919059 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:57.919021 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:58.416694 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:58.416663 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:59.626526 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:55:59.626491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:01.410564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:01.410518 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:02.992429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:02.992351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:04.670954 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:04.670919 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:06.312843 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:06.312807 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:07.998837 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:07.998797 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:09.595822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:09.595774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:11.126363 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:11.126307 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:12.865527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:12.865490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:14.521241 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:14.521205 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:15.883265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.883196 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf"] Oct 30 21:56:15.887328 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.887307 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.903548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.903527 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c2rf5\"" Oct 30 21:56:15.918422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.918367 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf"] Oct 30 21:56:15.963783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.963755 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.963917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.963788 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.963917 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.963815 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7z85\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.963996 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.963960 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.964109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.964000 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.964109 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.964025 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.964201 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.964175 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.964382 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.964359 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:15.964477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:15.964445 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.065939 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.065905 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.065964 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.065995 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066021 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066071 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066054 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7z85\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066132 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066179 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066230 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066350 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066675 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066646 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066728 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066828 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066803 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.066953 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.066867 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.067080 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.067060 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.068626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.068603 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.068768 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.068743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:16.068984 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.068966 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.083032 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.083004 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.083110 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.083048 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7z85\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85\") pod \"maas-default-gateway-openshift-default-77f556d647-z8jbf\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.202297 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.202269 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:16.229728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.229646 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:16.509872 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.509846 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf"] Oct 30 21:56:16.510460 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:56:16.510435 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cc53e2_1f21_48a7_b95a_4d0aab2f3d76.slice/crio-ea1408df77be26910b0bf4cb75dc58913448f9e85f7daae7f701b1f1874674ae WatchSource:0}: Error finding container ea1408df77be26910b0bf4cb75dc58913448f9e85f7daae7f701b1f1874674ae: Status 404 returned error can't find the container with id ea1408df77be26910b0bf4cb75dc58913448f9e85f7daae7f701b1f1874674ae Oct 30 21:56:16.512912 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.512877 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:16.513012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.512943 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:16.513012 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.512972 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:16.992970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.992937 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" event={"ID":"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76","Type":"ContainerStarted","Data":"1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182"} Oct 30 21:56:16.993364 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:16.992987 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" event={"ID":"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76","Type":"ContainerStarted","Data":"ea1408df77be26910b0bf4cb75dc58913448f9e85f7daae7f701b1f1874674ae"} Oct 30 21:56:17.052998 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:17.052946 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podStartSLOduration=2.052933115 podStartE2EDuration="2.052933115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:17.047739476 +0000 UTC m=+2816.257679104" watchObservedRunningTime="2026-10-30 21:56:17.052933115 +0000 UTC m=+2816.262872743" Oct 30 21:56:17.202767 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:17.202740 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:17.203845 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:17.203821 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:17.203963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:17.203864 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:17.911299 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:17.911257 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:18.203178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:18.203089 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:18.203178 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:18.203154 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:19.202798 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:19.202762 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:19.202981 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:19.202818 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:19.784282 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:19.784244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:20.203270 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:20.203233 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:20.203466 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:20.203292 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:20.484127 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:20.484051 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:56:21.214313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:21.213861 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:21.214313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:21.213933 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:21.900411 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:21.900280 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:22.203455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:22.203368 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:22.203455 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:22.203435 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:23.202576 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:23.202539 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-z8jbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:56:23.203031 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:23.202595 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:56:24.101834 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:24.101802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:24.926496 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:24.926458 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:24.927278 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:24.927215 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:24.928258 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:24.928226 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 21:56:25.188572 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:56:25.188491 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:56:26.312938 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:26.312897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:27.972155 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:27.972110 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:29.906791 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:29.906751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:30.970822 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:30.970789 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t"] Oct 30 21:56:30.976085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:30.976063 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.010976 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.010950 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t"] Oct 30 21:56:31.014704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.014681 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn"] Oct 30 21:56:31.019723 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.019699 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.043627 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.043603 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn"] Oct 30 21:56:31.146198 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146169 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146311 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146215 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqpbg\" (UniqueName: \"kubernetes.io/projected/b60c92be-c745-435e-b16b-6caf9000e769-kube-api-access-dqpbg\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146311 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146238 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc2e906-5a26-491a-bc4a-898e3a323816-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146383 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146354 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146421 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146449 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146491 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146473 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146533 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjrz\" (UniqueName: \"kubernetes.io/projected/adc2e906-5a26-491a-bc4a-898e3a323816-kube-api-access-5vjrz\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146564 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146562 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146658 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146600 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc2e906-5a26-491a-bc4a-898e3a323816-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146696 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146685 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60c92be-c745-435e-b16b-6caf9000e769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146720 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc2e906-5a26-491a-bc4a-898e3a323816-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146755 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146772 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146889 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.146936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146910 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.146936 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146931 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60c92be-c745-435e-b16b-6caf9000e769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.147039 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.146956 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60c92be-c745-435e-b16b-6caf9000e769-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.247969 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248041 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248015 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248049 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248073 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248123 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjrz\" (UniqueName: \"kubernetes.io/projected/adc2e906-5a26-491a-bc4a-898e3a323816-kube-api-access-5vjrz\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248165 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248226 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc2e906-5a26-491a-bc4a-898e3a323816-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248267 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60c92be-c745-435e-b16b-6caf9000e769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248316 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc2e906-5a26-491a-bc4a-898e3a323816-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248375 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248404 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248423 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248433 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248491 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248639 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248647 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248671 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.248737 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248708 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60c92be-c745-435e-b16b-6caf9000e769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248748 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60c92be-c745-435e-b16b-6caf9000e769-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248793 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248809 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248839 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqpbg\" (UniqueName: \"kubernetes.io/projected/b60c92be-c745-435e-b16b-6caf9000e769-kube-api-access-dqpbg\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.248965 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248855 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc2e906-5a26-491a-bc4a-898e3a323816-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.249256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.248986 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc2e906-5a26-491a-bc4a-898e3a323816-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.249256 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.249200 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.249487 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.249464 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.249704 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.249682 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.249783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.249725 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b60c92be-c745-435e-b16b-6caf9000e769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.250856 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.250827 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b60c92be-c745-435e-b16b-6caf9000e769-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.251106 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.251088 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b60c92be-c745-435e-b16b-6caf9000e769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.251171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.251089 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc2e906-5a26-491a-bc4a-898e3a323816-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.251171 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.251095 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc2e906-5a26-491a-bc4a-898e3a323816-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.266289 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.266264 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqpbg\" (UniqueName: \"kubernetes.io/projected/b60c92be-c745-435e-b16b-6caf9000e769-kube-api-access-dqpbg\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.266633 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.266613 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjrz\" (UniqueName: \"kubernetes.io/projected/adc2e906-5a26-491a-bc4a-898e3a323816-kube-api-access-5vjrz\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.266860 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.266836 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b60c92be-c745-435e-b16b-6caf9000e769-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-m9s5t\" (UID: \"b60c92be-c745-435e-b16b-6caf9000e769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.267013 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.266993 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc2e906-5a26-491a-bc4a-898e3a323816-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zj6zn\" (UID: \"adc2e906-5a26-491a-bc4a-898e3a323816\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.290440 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.290422 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:31.334053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.334030 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:31.596867 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.596782 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:31.622233 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.622200 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t"] Oct 30 21:56:31.625677 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.625640 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:31.625784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.625714 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:31.625784 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.625751 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:31.657374 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.657351 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn"] Oct 30 21:56:31.659577 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:56:31.659554 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc2e906_5a26_491a_bc4a_898e3a323816.slice/crio-6369a34c545bad5aab46550e93939063148b9c9e5d79402020c6db4a422fb505 WatchSource:0}: Error finding container 6369a34c545bad5aab46550e93939063148b9c9e5d79402020c6db4a422fb505: Status 404 returned error can't find the container with id 6369a34c545bad5aab46550e93939063148b9c9e5d79402020c6db4a422fb505 Oct 30 21:56:31.662060 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.662025 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:31.662162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.662095 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:31.662162 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:31.662142 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:56:32.177559 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.177521 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" event={"ID":"b60c92be-c745-435e-b16b-6caf9000e769","Type":"ContainerStarted","Data":"3223495b1f6cb7b1eee62a43df1b6d22a1d97dd9a99263b0d5735a245ffe16d9"} Oct 30 21:56:32.177998 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.177581 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" event={"ID":"b60c92be-c745-435e-b16b-6caf9000e769","Type":"ContainerStarted","Data":"ccd43ce2066fddc6e7b76601f5ecf4ccaff2fe88c806e91099e160aeb7c9826a"} Oct 30 21:56:32.179441 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.179412 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" event={"ID":"adc2e906-5a26-491a-bc4a-898e3a323816","Type":"ContainerStarted","Data":"18f914ff00dd6cd7d0d6b63ea72a16235ec1ad8b3e0a17e470e4228dfa702e28"} Oct 30 21:56:32.179550 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.179467 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" event={"ID":"adc2e906-5a26-491a-bc4a-898e3a323816","Type":"ContainerStarted","Data":"6369a34c545bad5aab46550e93939063148b9c9e5d79402020c6db4a422fb505"} Oct 30 21:56:32.220703 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.220653 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podStartSLOduration=2.220641614 podStartE2EDuration="2.220641614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:32.217812799 +0000 UTC m=+2831.427752427" watchObservedRunningTime="2026-10-30 21:56:32.220641614 +0000 UTC m=+2831.430581242" Oct 30 21:56:32.257866 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.257825 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podStartSLOduration=2.257811774 podStartE2EDuration="2.257811774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:32.254807638 +0000 UTC m=+2831.464747266" watchObservedRunningTime="2026-10-30 21:56:32.257811774 +0000 UTC m=+2831.467751402" Oct 30 21:56:32.291053 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.291030 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:32.292734 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.292713 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:32.292824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.292751 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:32.334773 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.334756 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:32.335992 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.335971 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:32.336066 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:32.336014 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:33.304418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:33.303943 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:33.304418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:33.303998 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:33.319582 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:33.319550 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:33.334428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:33.334371 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:33.334581 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:33.334430 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:34.301415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:34.299689 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:34.301415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:34.299762 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:34.334513 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:34.334487 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:34.334859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:34.334528 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:35.022775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:35.022737 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:35.293483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:35.292897 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:35.293483 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:35.292957 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:35.335188 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:35.335154 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:35.335626 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:35.335197 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:36.297414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:36.296663 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:36.297414 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:36.296719 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:36.335167 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:36.335138 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:36.335337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:36.335181 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:36.602081 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:36.601987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:36.768412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:36.766082 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:37.076958 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:37.076921 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:37.301819 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:37.301780 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:37.303267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:37.303233 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:37.334932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:37.334871 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:37.334932 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:37.334915 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:38.290676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:38.290645 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:38.291118 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:38.290697 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:38.334632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:38.334603 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zj6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:56:38.334759 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:38.334658 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" podUID="adc2e906-5a26-491a-bc4a-898e3a323816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 21:56:38.994340 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:38.994096 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:39.291144 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:39.291062 5039 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-m9s5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:56:39.291144 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:39.291122 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" podUID="b60c92be-c745-435e-b16b-6caf9000e769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 21:56:40.111951 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.111908 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:40.112431 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.112369 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:40.113537 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.113514 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zj6zn" Oct 30 21:56:40.295859 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.295827 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:40.296469 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.296301 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:40.297244 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.297223 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-m9s5t" Oct 30 21:56:40.785987 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:40.785939 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:42.526422 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:42.526337 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.092542 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:44.090736 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.537009 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:44.536975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.683820 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:44.683784 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:45.804096 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:45.804036 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:47.514237 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:47.514197 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:49.213182 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:49.213149 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:49.544333 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:49.544234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:49.704450 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:49.704413 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.911227 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:50.911195 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:51.091770 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:51.091733 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:52.616129 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:52.616092 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:54.249799 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:54.249760 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:55.185713 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:56:55.185677 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:56:56.094775 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:56.094734 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:57.598094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:57.598055 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.126488 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:59.126410 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.948330 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:56:59.948299 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:57:00.790410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:00.785923 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:01.585841 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:01.585781 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:01.935762 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:01.935727 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:02.093498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:02.093465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:03.100474 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:03.100433 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:03.256829 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:03.256375 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:03.715216 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:03.715179 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:04.920234 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:04.920085 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:05.737349 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:05.737306 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:06.491833 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:06.491780 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:08.033553 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:08.033511 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.670170 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:09.669642 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.867056 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:09.867020 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:10.033815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:10.033735 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:11.317529 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:11.317495 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:12.951069 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:12.951012 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:14.642588 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:14.642543 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:16.107292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:16.107260 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:16.280267 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:16.280231 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:16.435091 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:16.435056 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:17.029010 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:17.028972 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:17.186228 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:17.186191 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:18.391361 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:18.391321 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:19.996732 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:19.996700 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:21.669021 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:21.668984 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:23.455051 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:23.455017 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.186751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:25.186714 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.200529 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:57:25.200477 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:57:25.647863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:25.647825 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.813046 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:25.813004 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:26.714647 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:26.714606 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:26.870964 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:26.870929 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:27.028321 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:27.028246 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:27.347020 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:27.346928 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:28.702630 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:28.702596 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:30.371498 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:30.371379 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:32.088594 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:32.088551 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.145335 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:34.145304 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.336424 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:34.335369 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:36.184548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:36.184506 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:37.778154 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:37.778118 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:39.327130 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:39.327084 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:41.101560 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:41.101521 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:41.272477 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:41.272445 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:42.143893 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:42.143859 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:42.325082 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:42.325046 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:43.158678 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:43.158627 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:44.878351 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:44.878316 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:46.600317 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:46.600271 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:48.421717 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:48.421677 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:48.574596 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:48.574560 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:48.735218 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:48.735136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:50.069292 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:50.069253 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:50.655077 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:50.655039 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.129265 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:51.129175 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.312423 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:51.312303 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.486313 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:51.486279 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.685277 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:51.685244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.838421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:51.838330 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.380377 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:52.380333 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.520934 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:52.520896 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.837997 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:52.837945 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.584355 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:53.584326 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:55.027436 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:55.027373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:55.185259 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:57:55.185225 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:57:56.293782 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:56.293744 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:56.453527 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:56.453490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:56.699416 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:56.699364 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:57.356638 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:57.356597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:57.517428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:57.506796 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:57.772850 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:57.772816 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:57.926863 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:57.926828 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:58.246716 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:58.246683 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:58.402274 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:58.402234 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:58.619753 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:57:58.619678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:00.303062 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:00.303025 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:01.816793 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:01.816725 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:03.432941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:03.432853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:04.368686 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:04.368651 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:04.496517 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:04.496486 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:58:04.530728 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:04.530696 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:04.930199 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:04.930173 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:05.067727 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:05.067688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:05.420040 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:05.420009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:05.864337 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:05.864237 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:06.019815 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:06.019777 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:07.783538 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:07.783496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:08.297413 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.295612 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d"] Oct 30 21:58:08.304428 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.301314 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.317338 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.317314 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:58:08.317586 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.317568 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:58:08.319495 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.319477 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:58:08.338240 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.338217 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l5v2k\"" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.341894 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.341948 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.341974 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng88s\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.342084 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.342112 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.342156 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.344433 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.342211 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.349676 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.349656 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d"] Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.445705 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.445757 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.445807 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.445870 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.446025 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.446065 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.446429 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.446090 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng88s\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.450222 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.450195 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.455427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.455112 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.455427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.455286 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.455427 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.455347 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.456133 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.456085 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.525143 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.525047 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng88s\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.537931 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.537864 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4vc5d\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:08.651123 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:08.651094 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:09.127105 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:09.127018 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d"] Oct 30 21:58:09.136136 ip-10-0-133-149 kubenswrapper[5039]: W1030 21:58:09.136110 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c3ef72_9567_4843_983e_6b72f95ab14a.slice/crio-25a0dbe44fbbf5de369500e503dc279c56f9ca73d52124596de241b9237fa565 WatchSource:0}: Error finding container 25a0dbe44fbbf5de369500e503dc279c56f9ca73d52124596de241b9237fa565: Status 404 returned error can't find the container with id 25a0dbe44fbbf5de369500e503dc279c56f9ca73d52124596de241b9237fa565 Oct 30 21:58:09.139824 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:09.139772 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:09.139920 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:09.139900 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:09.965548 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:09.965503 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" event={"ID":"74c3ef72-9567-4843-983e-6b72f95ab14a","Type":"ContainerStarted","Data":"d2badeb7112f40bc0b3398333d2649420f3a1f64d380d9efb7a5407b939ae672"} Oct 30 21:58:09.965747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:09.965569 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" event={"ID":"74c3ef72-9567-4843-983e-6b72f95ab14a","Type":"ContainerStarted","Data":"25a0dbe44fbbf5de369500e503dc279c56f9ca73d52124596de241b9237fa565"} Oct 30 21:58:09.965747 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:09.965607 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:10.035792 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:10.035733 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" podStartSLOduration=2.035719456 podStartE2EDuration="2.035719456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:10.022834203 +0000 UTC m=+2929.232773832" watchObservedRunningTime="2026-10-30 21:58:10.035719456 +0000 UTC m=+2929.245659084" Oct 30 21:58:10.175461 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:10.175372 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:10.573946 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:10.573908 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:10.736164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:10.736132 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:10.973783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:10.973757 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 21:58:11.534038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:11.534002 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:11.702418 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:11.702352 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:11.998578 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:11.998543 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:13.925160 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:13.924351 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:15.670783 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:15.670751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:16.280614 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:16.280576 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:17.385970 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:17.385669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:18.994072 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:18.994030 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.679272 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:20.679224 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.274406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:22.274358 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:23.895098 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:23.895055 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:25.084957 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:25.084923 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:25.184291 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:58:25.184260 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:58:25.546544 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:25.546513 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:27.101622 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:27.096246 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:28.621976 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:28.621942 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.293356 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:30.293312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.925473 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:31.925400 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:33.493659 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:33.493619 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:35.438093 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:35.438057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:37.239231 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:37.239191 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.518565 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:39.518519 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.099754 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:40.099702 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.249742 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:40.249705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:41.612419 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:41.612365 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:41.788410 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:41.787503 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:43.317993 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:43.317957 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:44.886811 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:44.886774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:46.410305 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:46.410268 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:48.118484 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:48.118453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:49.704200 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:49.704152 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:51.255881 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:51.255800 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:52.792085 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:52.792050 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:54.491332 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:54.491230 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:55.185099 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:58:55.185063 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:58:56.095802 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:56.095772 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:57.643421 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:57.643371 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:59.203925 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:58:59.203891 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:00.807545 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:00.807509 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:02.381556 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:02.381519 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:03.151563 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:03.151534 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:59:04.174963 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:04.174926 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:05.780094 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:05.780040 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:07.341692 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:07.341657 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:08.916961 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:08.916358 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:10.497164 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:10.497125 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.186597 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:12.186563 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:13.783097 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:13.783063 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:15.328100 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:15.328066 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:16.399888 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:16.399853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:17.001583 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:17.001552 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:18.689901 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:18.689864 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:20.204033 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:20.204001 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:21.379406 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:21.379366 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:59:21.391751 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:21.391728 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 21:59:21.887254 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:21.887223 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:23.460038 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:23.459842 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:25.012873 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:25.012759 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:25.184854 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:59:25.184823 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:59:26.557602 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:26.556910 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.094621 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:28.094581 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:29.610412 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:29.610259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:31.213955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:31.213922 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:32.899864 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:32.899774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:34.549645 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:34.549611 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:36.175193 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:36.175150 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:37.882058 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:37.877609 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:39.400073 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:39.400043 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:41.005632 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:41.005585 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:42.589281 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:42.589244 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:44.151196 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:44.151158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:45.819609 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:45.819578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:46.457794 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:46.457759 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 21:59:47.535213 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:47.535179 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:49.214092 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:49.214049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:50.744288 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:50.743579 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:52.381359 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:52.381319 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:54.005941 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:54.005907 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:55.185408 ip-10-0-133-149 kubenswrapper[5039]: E1030 21:59:55.185361 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 21:59:55.795756 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:55.795609 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:57.409955 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:57.409917 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:58.989415 ip-10-0-133-149 kubenswrapper[5039]: I1030 21:59:58.988794 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:00.262471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.262429 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z"] Oct 30 22:00:00.274944 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.274915 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5"] Oct 30 22:00:00.281378 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.275211 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.289425 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.289404 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl"] Oct 30 22:00:00.289797 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.289777 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.318591 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.318565 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z"] Oct 30 22:00:00.318736 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.318614 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5"] Oct 30 22:00:00.318996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.318640 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl"] Oct 30 22:00:00.319081 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.318909 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.373206 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.373184 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\"" Oct 30 22:00:00.382917 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.382891 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.383077 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.383058 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8thm\" (UniqueName: \"kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.383166 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.383151 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vm6d\" (UniqueName: \"kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.383213 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.383205 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.484954 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.484920 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8thm\" (UniqueName: \"kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.485101 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.484959 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2msn\" (UniqueName: \"kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.485101 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.485004 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vm6d\" (UniqueName: \"kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.485101 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.485044 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.485341 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.485318 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.485464 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.485440 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.485739 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.485722 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.485947 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.485932 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.504834 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.504811 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vm6d\" (UniqueName: \"kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.506095 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.506072 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8thm\" (UniqueName: \"kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.586778 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.586725 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2msn\" (UniqueName: \"kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.586891 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.586872 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.587423 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.587405 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.603811 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.603782 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2msn\" (UniqueName: \"kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.609894 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.609693 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:00.614687 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.614662 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:00.627123 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.623527 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:00.633913 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.633887 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:00.936974 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:00:00.936947 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3852928_e949_4388_b4a6_950f9cf70f21.slice/crio-f87e7f0ab81db0ae8aeff7b55e95142d2845d588c2cdeff5b61390971e87b4cc WatchSource:0}: Error finding container f87e7f0ab81db0ae8aeff7b55e95142d2845d588c2cdeff5b61390971e87b4cc: Status 404 returned error can't find the container with id f87e7f0ab81db0ae8aeff7b55e95142d2845d588c2cdeff5b61390971e87b4cc Oct 30 22:00:00.937105 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.937055 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z"] Oct 30 22:00:00.962720 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.962695 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5"] Oct 30 22:00:00.965974 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:00.965953 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl"] Oct 30 22:00:00.968692 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:00:00.968671 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877c5621_c54f_4c04_91f0_06655af72601.slice/crio-9e9c29c64203f6d2540fa588c8392e7339922747e1ddc37f61edc8450b8bdcc2 WatchSource:0}: Error finding container 9e9c29c64203f6d2540fa588c8392e7339922747e1ddc37f61edc8450b8bdcc2: Status 404 returned error can't find the container with id 9e9c29c64203f6d2540fa588c8392e7339922747e1ddc37f61edc8450b8bdcc2 Oct 30 22:00:00.968892 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:00:00.968875 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a29e410_fd4c_4016_aa8b_3e7b678b1967.slice/crio-59b20f5932458d18d0cdef8dbe38140d26a8fd6d1843ea883073621a4a27315a WatchSource:0}: Error finding container 59b20f5932458d18d0cdef8dbe38140d26a8fd6d1843ea883073621a4a27315a: Status 404 returned error can't find the container with id 59b20f5932458d18d0cdef8dbe38140d26a8fd6d1843ea883073621a4a27315a Oct 30 22:00:01.753911 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.753870 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" event={"ID":"1a29e410-fd4c-4016-aa8b-3e7b678b1967","Type":"ContainerStarted","Data":"972d69ba8e462681b086e68a5ad54f1317994f37f3470f31ec7f5e20fee77b7a"} Oct 30 22:00:01.754345 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.753956 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" event={"ID":"1a29e410-fd4c-4016-aa8b-3e7b678b1967","Type":"ContainerStarted","Data":"59b20f5932458d18d0cdef8dbe38140d26a8fd6d1843ea883073621a4a27315a"} Oct 30 22:00:01.755566 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.755534 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" event={"ID":"877c5621-c54f-4c04-91f0-06655af72601","Type":"ContainerStarted","Data":"3f7a58700f3125dd6d33f0283bded544b2c9bb3895e874d5b1eed75c87606918"} Oct 30 22:00:01.755660 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.755577 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" event={"ID":"877c5621-c54f-4c04-91f0-06655af72601","Type":"ContainerStarted","Data":"9e9c29c64203f6d2540fa588c8392e7339922747e1ddc37f61edc8450b8bdcc2"} Oct 30 22:00:01.757028 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.757006 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" event={"ID":"f3852928-e949-4388-b4a6-950f9cf70f21","Type":"ContainerStarted","Data":"4f1d666b6f9ed8b606edd6d3860d41498080107c8ecb71489bfb0007f5bd2691"} Oct 30 22:00:01.757126 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.757044 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" event={"ID":"f3852928-e949-4388-b4a6-950f9cf70f21","Type":"ContainerStarted","Data":"f87e7f0ab81db0ae8aeff7b55e95142d2845d588c2cdeff5b61390971e87b4cc"} Oct 30 22:00:01.788018 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.787977 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" podStartSLOduration=1.787962452 podStartE2EDuration="1.787962452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.78720337 +0000 UTC m=+3040.997142999" watchObservedRunningTime="2026-10-30 22:00:01.787962452 +0000 UTC m=+3040.997902082" Oct 30 22:00:01.822895 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.822855 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" podStartSLOduration=1.8228442519999999 podStartE2EDuration="1.822844252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.819547736 +0000 UTC m=+3041.029487364" watchObservedRunningTime="2026-10-30 22:00:01.822844252 +0000 UTC m=+3041.032783880" Oct 30 22:00:01.854035 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:01.853995 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" podStartSLOduration=1.8539843550000001 podStartE2EDuration="1.853984355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.851790069 +0000 UTC m=+3041.061729697" watchObservedRunningTime="2026-10-30 22:00:01.853984355 +0000 UTC m=+3041.063923983" Oct 30 22:00:02.300372 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.299828 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:02.507206 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.507164 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:02.655666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.655629 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:02.765315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.765232 5039 generic.go:356] "Generic (PLEG): container finished" podID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerID="972d69ba8e462681b086e68a5ad54f1317994f37f3470f31ec7f5e20fee77b7a" exitCode=60 Oct 30 22:00:02.765315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.765301 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" event={"ID":"1a29e410-fd4c-4016-aa8b-3e7b678b1967","Type":"ContainerDied","Data":"972d69ba8e462681b086e68a5ad54f1317994f37f3470f31ec7f5e20fee77b7a"} Oct 30 22:00:02.765803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.765630 5039 scope.go:122] "RemoveContainer" containerID="972d69ba8e462681b086e68a5ad54f1317994f37f3470f31ec7f5e20fee77b7a" Oct 30 22:00:02.767079 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.767049 5039 generic.go:356] "Generic (PLEG): container finished" podID="877c5621-c54f-4c04-91f0-06655af72601" containerID="3f7a58700f3125dd6d33f0283bded544b2c9bb3895e874d5b1eed75c87606918" exitCode=60 Oct 30 22:00:02.767197 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.767125 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" event={"ID":"877c5621-c54f-4c04-91f0-06655af72601","Type":"ContainerDied","Data":"3f7a58700f3125dd6d33f0283bded544b2c9bb3895e874d5b1eed75c87606918"} Oct 30 22:00:02.767399 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.767373 5039 scope.go:122] "RemoveContainer" containerID="3f7a58700f3125dd6d33f0283bded544b2c9bb3895e874d5b1eed75c87606918" Oct 30 22:00:02.768991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.768878 5039 generic.go:356] "Generic (PLEG): container finished" podID="f3852928-e949-4388-b4a6-950f9cf70f21" containerID="4f1d666b6f9ed8b606edd6d3860d41498080107c8ecb71489bfb0007f5bd2691" exitCode=60 Oct 30 22:00:02.768991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.768905 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" event={"ID":"f3852928-e949-4388-b4a6-950f9cf70f21","Type":"ContainerDied","Data":"4f1d666b6f9ed8b606edd6d3860d41498080107c8ecb71489bfb0007f5bd2691"} Oct 30 22:00:02.769259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:02.769214 5039 scope.go:122] "RemoveContainer" containerID="4f1d666b6f9ed8b606edd6d3860d41498080107c8ecb71489bfb0007f5bd2691" Oct 30 22:00:03.778386 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.778359 5039 generic.go:356] "Generic (PLEG): container finished" podID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerID="112d2b30c074c184e73f1181ad7461f653ebfa81a7eec50e9fa7c53a50ab21f3" exitCode=60 Oct 30 22:00:03.778791 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.778430 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" event={"ID":"1a29e410-fd4c-4016-aa8b-3e7b678b1967","Type":"ContainerDied","Data":"112d2b30c074c184e73f1181ad7461f653ebfa81a7eec50e9fa7c53a50ab21f3"} Oct 30 22:00:03.778791 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.778511 5039 scope.go:122] "RemoveContainer" containerID="972d69ba8e462681b086e68a5ad54f1317994f37f3470f31ec7f5e20fee77b7a" Oct 30 22:00:03.778791 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.778740 5039 scope.go:122] "RemoveContainer" containerID="112d2b30c074c184e73f1181ad7461f653ebfa81a7eec50e9fa7c53a50ab21f3" Oct 30 22:00:03.779129 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:03.779104 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl_odh-ai-gateway-infra(1a29e410-fd4c-4016-aa8b-3e7b678b1967)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" Oct 30 22:00:03.781263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.781242 5039 generic.go:356] "Generic (PLEG): container finished" podID="877c5621-c54f-4c04-91f0-06655af72601" containerID="d53ab4fc2dfe1d68bcb30ac34a88db0e92be1ef16b8c2ce59b2a2b7491d4c0f8" exitCode=60 Oct 30 22:00:03.781352 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.781287 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" event={"ID":"877c5621-c54f-4c04-91f0-06655af72601","Type":"ContainerDied","Data":"d53ab4fc2dfe1d68bcb30ac34a88db0e92be1ef16b8c2ce59b2a2b7491d4c0f8"} Oct 30 22:00:03.781602 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.781575 5039 scope.go:122] "RemoveContainer" containerID="d53ab4fc2dfe1d68bcb30ac34a88db0e92be1ef16b8c2ce59b2a2b7491d4c0f8" Oct 30 22:00:03.781905 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:03.781881 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5_odh-ai-gateway-infra(877c5621-c54f-4c04-91f0-06655af72601)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" podUID="877c5621-c54f-4c04-91f0-06655af72601" Oct 30 22:00:03.783282 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.783261 5039 generic.go:356] "Generic (PLEG): container finished" podID="f3852928-e949-4388-b4a6-950f9cf70f21" containerID="41e6706c9364315ce1a0da04f4702dd4648ed0d0437a89ffbc72f9ba83c8dde8" exitCode=60 Oct 30 22:00:03.783378 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.783335 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" event={"ID":"f3852928-e949-4388-b4a6-950f9cf70f21","Type":"ContainerDied","Data":"41e6706c9364315ce1a0da04f4702dd4648ed0d0437a89ffbc72f9ba83c8dde8"} Oct 30 22:00:03.783550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.783537 5039 scope.go:122] "RemoveContainer" containerID="41e6706c9364315ce1a0da04f4702dd4648ed0d0437a89ffbc72f9ba83c8dde8" Oct 30 22:00:03.783771 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:03.783755 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z_odh-ai-gateway-infra(f3852928-e949-4388-b4a6-950f9cf70f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" Oct 30 22:00:03.905960 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.905939 5039 scope.go:122] "RemoveContainer" containerID="3f7a58700f3125dd6d33f0283bded544b2c9bb3895e874d5b1eed75c87606918" Oct 30 22:00:03.918593 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:03.918576 5039 scope.go:122] "RemoveContainer" containerID="4f1d666b6f9ed8b606edd6d3860d41498080107c8ecb71489bfb0007f5bd2691" Oct 30 22:00:04.312369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:04.311685 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:04.792071 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:04.792044 5039 scope.go:122] "RemoveContainer" containerID="112d2b30c074c184e73f1181ad7461f653ebfa81a7eec50e9fa7c53a50ab21f3" Oct 30 22:00:04.792489 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:04.792311 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl_odh-ai-gateway-infra(1a29e410-fd4c-4016-aa8b-3e7b678b1967)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" Oct 30 22:00:04.793602 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:04.793581 5039 scope.go:122] "RemoveContainer" containerID="d53ab4fc2dfe1d68bcb30ac34a88db0e92be1ef16b8c2ce59b2a2b7491d4c0f8" Oct 30 22:00:04.793826 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:04.793804 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5_odh-ai-gateway-infra(877c5621-c54f-4c04-91f0-06655af72601)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" podUID="877c5621-c54f-4c04-91f0-06655af72601" Oct 30 22:00:04.795106 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:04.795090 5039 scope.go:122] "RemoveContainer" containerID="41e6706c9364315ce1a0da04f4702dd4648ed0d0437a89ffbc72f9ba83c8dde8" Oct 30 22:00:04.795298 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:04.795280 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z_odh-ai-gateway-infra(f3852928-e949-4388-b4a6-950f9cf70f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" Oct 30 22:00:06.042769 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:06.041853 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:07.706487 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:07.706453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.693894 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:08.693861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:09.455121 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:09.455074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:11.075842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:11.075804 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:13.876114 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:13.876080 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:14.207210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:14.207159 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:14.667121 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:14.667093 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:14.806410 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:14.805547 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:14.914766 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:14.914739 5039 scope.go:122] "RemoveContainer" containerID="d53ab4fc2dfe1d68bcb30ac34a88db0e92be1ef16b8c2ce59b2a2b7491d4c0f8" Oct 30 22:00:15.089832 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.089805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:15.871694 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.871670 5039 generic.go:356] "Generic (PLEG): container finished" podID="877c5621-c54f-4c04-91f0-06655af72601" containerID="499ebfd6d527a90c44440f3c808fea71eb909b02017850ef458fdaa3f5a16961" exitCode=60 Oct 30 22:00:15.871832 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.871749 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" event={"ID":"877c5621-c54f-4c04-91f0-06655af72601","Type":"ContainerDied","Data":"499ebfd6d527a90c44440f3c808fea71eb909b02017850ef458fdaa3f5a16961"} Oct 30 22:00:15.871832 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.871817 5039 scope.go:122] "RemoveContainer" containerID="d53ab4fc2dfe1d68bcb30ac34a88db0e92be1ef16b8c2ce59b2a2b7491d4c0f8" Oct 30 22:00:15.872119 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.872098 5039 scope.go:122] "RemoveContainer" containerID="499ebfd6d527a90c44440f3c808fea71eb909b02017850ef458fdaa3f5a16961" Oct 30 22:00:15.872373 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:15.872350 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5_odh-ai-gateway-infra(877c5621-c54f-4c04-91f0-06655af72601)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" podUID="877c5621-c54f-4c04-91f0-06655af72601" Oct 30 22:00:15.914284 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.914264 5039 scope.go:122] "RemoveContainer" containerID="41e6706c9364315ce1a0da04f4702dd4648ed0d0437a89ffbc72f9ba83c8dde8" Oct 30 22:00:15.934069 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:15.934006 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:16.507533 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.507462 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:16.883035 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.883001 5039 generic.go:356] "Generic (PLEG): container finished" podID="f3852928-e949-4388-b4a6-950f9cf70f21" containerID="a93156abe90ea91978c13972cfcfbc5ac230d7eb8d9bec5370566ba6979d1112" exitCode=60 Oct 30 22:00:16.883177 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.883071 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" event={"ID":"f3852928-e949-4388-b4a6-950f9cf70f21","Type":"ContainerDied","Data":"a93156abe90ea91978c13972cfcfbc5ac230d7eb8d9bec5370566ba6979d1112"} Oct 30 22:00:16.883177 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.883128 5039 scope.go:122] "RemoveContainer" containerID="41e6706c9364315ce1a0da04f4702dd4648ed0d0437a89ffbc72f9ba83c8dde8" Oct 30 22:00:16.883436 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.883416 5039 scope.go:122] "RemoveContainer" containerID="a93156abe90ea91978c13972cfcfbc5ac230d7eb8d9bec5370566ba6979d1112" Oct 30 22:00:16.883675 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:16.883648 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z_odh-ai-gateway-infra(f3852928-e949-4388-b4a6-950f9cf70f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" Oct 30 22:00:16.914176 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.914157 5039 scope.go:122] "RemoveContainer" containerID="112d2b30c074c184e73f1181ad7461f653ebfa81a7eec50e9fa7c53a50ab21f3" Oct 30 22:00:16.925052 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:16.925033 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5"] Oct 30 22:00:17.080294 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.080265 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:17.186985 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.186958 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca\") pod \"877c5621-c54f-4c04-91f0-06655af72601\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " Oct 30 22:00:17.187125 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.187034 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm\" (UniqueName: \"kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm\") pod \"877c5621-c54f-4c04-91f0-06655af72601\" (UID: \"877c5621-c54f-4c04-91f0-06655af72601\") " Oct 30 22:00:17.187274 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.187251 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca" pod "877c5621-c54f-4c04-91f0-06655af72601" (UID: "877c5621-c54f-4c04-91f0-06655af72601"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:17.187466 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.187453 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c5621-c54f-4c04-91f0-06655af72601-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:00:17.258046 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.258018 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm" pod "877c5621-c54f-4c04-91f0-06655af72601" (UID: "877c5621-c54f-4c04-91f0-06655af72601"). InnerVolumeSpecName "kube-api-access-k8thm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:17.288831 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.288798 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8thm\" (UniqueName: \"kubernetes.io/projected/877c5621-c54f-4c04-91f0-06655af72601-kube-api-access-k8thm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:00:17.587055 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.586706 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:17.891663 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.891633 5039 generic.go:356] "Generic (PLEG): container finished" podID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerID="98d357d3cbccb7730bd9422e4b32c8013e495f5843168162192db1befdc356fb" exitCode=60 Oct 30 22:00:17.891778 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.891707 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" event={"ID":"1a29e410-fd4c-4016-aa8b-3e7b678b1967","Type":"ContainerDied","Data":"98d357d3cbccb7730bd9422e4b32c8013e495f5843168162192db1befdc356fb"} Oct 30 22:00:17.891778 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.891766 5039 scope.go:122] "RemoveContainer" containerID="112d2b30c074c184e73f1181ad7461f653ebfa81a7eec50e9fa7c53a50ab21f3" Oct 30 22:00:17.892161 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.892143 5039 scope.go:122] "RemoveContainer" containerID="98d357d3cbccb7730bd9422e4b32c8013e495f5843168162192db1befdc356fb" Oct 30 22:00:17.892499 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:17.892469 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl_odh-ai-gateway-infra(1a29e410-fd4c-4016-aa8b-3e7b678b1967)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" Oct 30 22:00:17.893084 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.893066 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" Oct 30 22:00:17.893084 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.893077 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5" event={"ID":"877c5621-c54f-4c04-91f0-06655af72601","Type":"ContainerDied","Data":"9e9c29c64203f6d2540fa588c8392e7339922747e1ddc37f61edc8450b8bdcc2"} Oct 30 22:00:17.922952 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.922814 5039 scope.go:122] "RemoveContainer" containerID="499ebfd6d527a90c44440f3c808fea71eb909b02017850ef458fdaa3f5a16961" Oct 30 22:00:17.932707 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.932690 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z"] Oct 30 22:00:17.972277 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.972217 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5"] Oct 30 22:00:17.981550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:17.980801 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ae6492-29889960-cxtq5"] Oct 30 22:00:18.102089 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.102063 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:18.203811 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.203779 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d\" (UniqueName: \"kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d\") pod \"f3852928-e949-4388-b4a6-950f9cf70f21\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " Oct 30 22:00:18.203976 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.203963 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca\") pod \"f3852928-e949-4388-b4a6-950f9cf70f21\" (UID: \"f3852928-e949-4388-b4a6-950f9cf70f21\") " Oct 30 22:00:18.204347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.204322 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca" pod "f3852928-e949-4388-b4a6-950f9cf70f21" (UID: "f3852928-e949-4388-b4a6-950f9cf70f21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.204491 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.204457 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3852928-e949-4388-b4a6-950f9cf70f21-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:00:18.205903 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.205877 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d" pod "f3852928-e949-4388-b4a6-950f9cf70f21" (UID: "f3852928-e949-4388-b4a6-950f9cf70f21"). InnerVolumeSpecName "kube-api-access-5vm6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.305077 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.305054 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vm6d\" (UniqueName: \"kubernetes.io/projected/f3852928-e949-4388-b4a6-950f9cf70f21-kube-api-access-5vm6d\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:00:18.904443 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.904419 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" Oct 30 22:00:18.904588 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.904446 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z" event={"ID":"f3852928-e949-4388-b4a6-950f9cf70f21","Type":"ContainerDied","Data":"f87e7f0ab81db0ae8aeff7b55e95142d2845d588c2cdeff5b61390971e87b4cc"} Oct 30 22:00:18.904588 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.904533 5039 scope.go:122] "RemoveContainer" containerID="a93156abe90ea91978c13972cfcfbc5ac230d7eb8d9bec5370566ba6979d1112" Oct 30 22:00:18.923003 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.922980 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877c5621-c54f-4c04-91f0-06655af72601" path="/var/lib/kubelet/pods/877c5621-c54f-4c04-91f0-06655af72601/volumes" Oct 30 22:00:18.944570 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.944548 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl"] Oct 30 22:00:18.957973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.957950 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z"] Oct 30 22:00:18.964699 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:18.964668 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889960-2xm2z"] Oct 30 22:00:19.123192 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.123169 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:19.214735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.214649 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca\") pod \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " Oct 30 22:00:19.214900 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.214824 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn\" (UniqueName: \"kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn\") pod \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\" (UID: \"1a29e410-fd4c-4016-aa8b-3e7b678b1967\") " Oct 30 22:00:19.215079 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.215050 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca" pod "1a29e410-fd4c-4016-aa8b-3e7b678b1967" (UID: "1a29e410-fd4c-4016-aa8b-3e7b678b1967"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.215469 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.215444 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a29e410-fd4c-4016-aa8b-3e7b678b1967-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:00:19.217063 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.217039 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn" pod "1a29e410-fd4c-4016-aa8b-3e7b678b1967" (UID: "1a29e410-fd4c-4016-aa8b-3e7b678b1967"). InnerVolumeSpecName "kube-api-access-c2msn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.316279 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.316256 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2msn\" (UniqueName: \"kubernetes.io/projected/1a29e410-fd4c-4016-aa8b-3e7b678b1967-kube-api-access-c2msn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:00:19.444687 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.444644 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:19.915161 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.915132 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" Oct 30 22:00:19.915342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.915131 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl" event={"ID":"1a29e410-fd4c-4016-aa8b-3e7b678b1967","Type":"ContainerDied","Data":"59b20f5932458d18d0cdef8dbe38140d26a8fd6d1843ea883073621a4a27315a"} Oct 30 22:00:19.915342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.915282 5039 scope.go:122] "RemoveContainer" containerID="98d357d3cbccb7730bd9422e4b32c8013e495f5843168162192db1befdc356fb" Oct 30 22:00:19.961371 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.961342 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl"] Oct 30 22:00:19.974384 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:19.974363 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889960-jznwl"] Oct 30 22:00:20.923477 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:20.923445 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" path="/var/lib/kubelet/pods/1a29e410-fd4c-4016-aa8b-3e7b678b1967/volumes" Oct 30 22:00:20.924071 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:20.924052 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" path="/var/lib/kubelet/pods/f3852928-e949-4388-b4a6-950f9cf70f21/volumes" Oct 30 22:00:21.100114 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:21.100086 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:00:21.215181 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:21.215074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:22.853698 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:22.853659 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:22.995403 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:22.995340 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.160109 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:23.160074 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.337345 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:23.337312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.495161 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:23.495082 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.877838 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:23.877741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:24.258444 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:24.258407 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:24.801789 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:24.801747 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:25.031807 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:25.031766 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:25.184204 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:25.184167 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:25.192161 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:25.192123 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:00:26.649552 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:26.649515 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:26.920108 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:26.920005 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:27.622973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:27.622933 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:28.585547 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:28.585491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:30.156259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:30.156217 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:31.712526 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:31.712488 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:31.989411 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:31.989313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:32.138344 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:32.138303 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:33.794492 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:33.794451 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.407186 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:35.407143 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:37.008510 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:37.008462 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:38.299911 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:38.299874 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:38.530687 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:38.530658 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:40.098203 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:40.098168 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:41.806824 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:41.806788 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:42.485916 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:42.485880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:42.622048 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:42.622015 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:43.492063 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:43.492016 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:45.070134 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:45.070097 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:46.615290 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:46.615146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:48.191469 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:48.191430 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:49.812247 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:49.812213 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:51.412109 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:51.412071 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:53.106210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:53.106163 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:54.783311 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:54.783280 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:55.185530 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:00:55.185495 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:00:56.519313 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:56.519276 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.046706 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:58.046666 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:59.693086 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:00:59.693054 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.288272 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:01.288241 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:02.977443 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:02.975366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:04.524846 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:04.524784 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:06.151350 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:06.151293 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:06.914985 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:06.914952 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.073655 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:07.073615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.584477 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:07.584446 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:01:08.387299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:08.387258 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.541645 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:09.541607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.733515 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:09.733490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.910064 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:09.910033 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.166806 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:10.166715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.334620 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:10.334581 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.187427 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:12.187379 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.486037 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:12.485925 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.677910 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:12.677873 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.511300 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:13.511259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:14.249156 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:14.249099 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:14.541624 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:14.541546 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:16.017816 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:16.017778 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:16.635432 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:16.635367 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:17.636771 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:17.636729 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.160741 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:19.160685 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.314806 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:19.314755 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.494825 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:19.494747 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:20.797491 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:20.797456 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:22.409708 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:22.409663 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:24.017837 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:24.017798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:25.185036 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:01:25.185003 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:01:25.573581 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:25.573501 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:27.205579 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:27.205538 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:28.963409 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:28.963342 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.413267 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:31.413199 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.689050 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:31.688957 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:33.583531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:33.583481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:35.245689 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:35.245584 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:36.874022 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:36.873980 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:38.433828 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:38.433793 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:40.104776 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:40.104738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:41.658921 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:41.658873 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:43.289140 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:43.289089 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:44.932998 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:44.932965 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:46.719058 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:46.719020 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:48.485575 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:48.485530 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:50.106356 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:50.106319 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:51.736562 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:51.736524 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:53.315802 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:53.315759 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:54.899246 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:54.886959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:55.184445 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:01:55.184338 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:01:56.442020 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:56.441982 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.837453 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:57.837422 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:01:57.988146 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:57.988102 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:59.777727 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:01:59.777691 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:00.920610 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:00.920576 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:01.397706 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:01.397674 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:03.022190 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:03.022065 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:04.614281 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:04.614245 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:06.205886 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:06.205847 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:07.856649 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:07.856609 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:09.499728 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:09.499697 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:11.091657 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:11.091621 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:12.681278 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:12.681238 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.230432 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:14.230045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:15.821442 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:15.821358 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:16.741886 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:16.741848 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:17.608021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:17.607977 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:19.306493 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:19.306406 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:20.828551 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:20.828514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:22.487081 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:22.487041 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:24.113850 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:24.113805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:25.184970 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:02:25.184936 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:02:25.788541 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:25.788486 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:27.391868 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:27.391819 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:28.951072 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:28.951039 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:30.592199 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:30.592163 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.292775 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:32.292738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:33.816702 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:33.816669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:35.423953 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:35.423908 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:37.115351 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:37.115314 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:38.750260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:38.750221 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:40.413946 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:40.413909 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:42.116891 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:42.116846 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.726913 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:43.726877 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:45.295079 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:45.295033 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.450209 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:46.450173 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:02:46.925308 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:46.925271 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:48.503495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:48.503462 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:50.102888 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:50.102849 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:51.686018 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:51.685961 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:53.230328 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:53.230246 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.882992 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:54.882945 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.183941 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:02:55.183903 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:02:56.520623 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:56.520582 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:58.180056 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:58.180024 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:59.810337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:02:59.810298 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.475533 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:01.475501 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:03.123719 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:03.123678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:04.778355 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:04.778296 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:06.427597 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:06.427551 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.087871 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:08.087835 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:09.716301 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:09.716259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:11.390666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:11.390628 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:12.974794 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:12.974756 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:14.802245 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:14.802206 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:16.484421 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:16.483716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:16.874872 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:16.874795 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:18.100867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:18.100829 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:19.579648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:19.579619 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz"] Oct 30 22:03:19.580096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:19.579835 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" containerID="cri-o://62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab" gracePeriod=30 Oct 30 22:03:19.801296 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:19.801263 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.581775 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:21.581741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.692539 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:21.692505 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:03:23.111204 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:23.111160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:23.123307 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:23.123276 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:23.123510 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:23.123344 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:24.789762 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.789701 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:24.874037 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.874011 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 22:03:24.906937 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.906899 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907057 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.906968 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907057 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.906993 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907057 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907031 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907057 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907050 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907276 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907094 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907276 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907141 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907276 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907140 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:24.907276 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907192 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907276 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907247 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo\") pod \"57417a1b-d284-4aac-9a1d-f7843068d67e\" (UID: \"57417a1b-d284-4aac-9a1d-f7843068d67e\") " Oct 30 22:03:24.907559 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907320 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:24.907559 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907432 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:24.907559 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907522 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:24.907718 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.907621 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:24.908023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.908002 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:24.908023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.908023 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:24.908174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.908033 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:24.908174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.908041 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:24.908174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.908049 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57417a1b-d284-4aac-9a1d-f7843068d67e-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:24.909210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.909182 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "kube-api-access-hftb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:24.909286 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.909208 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:24.909647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.909628 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:24.909718 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:24.909641 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo" pod "57417a1b-d284-4aac-9a1d-f7843068d67e" (UID: "57417a1b-d284-4aac-9a1d-f7843068d67e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:25.009112 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.009080 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hftb2\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-kube-api-access-hftb2\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:25.009112 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.009107 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:25.009263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.009120 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:25.009263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.009132 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57417a1b-d284-4aac-9a1d-f7843068d67e-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:25.186200 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:03:25.186162 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:03:25.279538 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.279506 5039 generic.go:356] "Generic (PLEG): container finished" podID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerID="62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab" exitCode=0 Oct 30 22:03:25.279700 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.279549 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" Oct 30 22:03:25.279700 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.279575 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" event={"ID":"57417a1b-d284-4aac-9a1d-f7843068d67e","Type":"ContainerDied","Data":"62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab"} Oct 30 22:03:25.279700 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.279617 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz" event={"ID":"57417a1b-d284-4aac-9a1d-f7843068d67e","Type":"ContainerDied","Data":"121c3c0c05755c628775a1330cc4aadbcfd8982fd0e70ce929ef1114d43ed775"} Oct 30 22:03:25.279700 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.279638 5039 scope.go:122] "RemoveContainer" containerID="62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab" Oct 30 22:03:25.289749 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.289724 5039 scope.go:122] "RemoveContainer" containerID="62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab" Oct 30 22:03:25.290024 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:03:25.290005 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab\": container with ID starting with 62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab not found: ID does not exist" containerID="62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab" Oct 30 22:03:25.290071 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.290033 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab"} err="failed to get container status \"62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab\": rpc error: code = NotFound desc = could not find container \"62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab\": container with ID starting with 62c18f18f347443000944575576ba6dd99a9ec6fb6c16506e1cb3b2e0cf104ab not found: ID does not exist" Oct 30 22:03:25.310292 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.310273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:25.325232 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.325209 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz"] Oct 30 22:03:25.336090 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.336066 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4j6qz"] Oct 30 22:03:25.466204 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:25.466098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:26.796219 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:26.795084 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:26.923474 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:26.923436 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" path="/var/lib/kubelet/pods/57417a1b-d284-4aac-9a1d-f7843068d67e/volumes" Oct 30 22:03:27.620402 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:27.620349 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.779810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:27.779772 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:28.994606 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:28.994572 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:29.645962 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:29.645922 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:29.787177 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:29.787144 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.469106 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:30.469075 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.608075 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:30.608038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.948848 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:30.948818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.047576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:32.047537 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.209715 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:32.209669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.671332 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:32.671295 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.856994 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:32.856957 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:33.005321 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:33.005239 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:33.672862 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:33.672828 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:33.837906 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:33.837875 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:34.906352 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:34.906312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:35.059647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:35.059615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.519242 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:36.519196 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:38.135459 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:38.132066 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.145965 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.145931 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.241999 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.241970 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d"] Oct 30 22:03:39.242177 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.242158 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" podUID="74c3ef72-9567-4843-983e-6b72f95ab14a" containerName="discovery" containerID="cri-o://d2badeb7112f40bc0b3398333d2649420f3a1f64d380d9efb7a5407b939ae672" gracePeriod=30 Oct 30 22:03:39.314282 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.314249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.414971 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.414900 5039 generic.go:356] "Generic (PLEG): container finished" podID="74c3ef72-9567-4843-983e-6b72f95ab14a" containerID="d2badeb7112f40bc0b3398333d2649420f3a1f64d380d9efb7a5407b939ae672" exitCode=0 Oct 30 22:03:39.415097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.414971 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" event={"ID":"74c3ef72-9567-4843-983e-6b72f95ab14a","Type":"ContainerDied","Data":"d2badeb7112f40bc0b3398333d2649420f3a1f64d380d9efb7a5407b939ae672"} Oct 30 22:03:39.541898 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.541872 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 22:03:39.607619 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.607584 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.641270 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641243 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641410 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641321 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641469 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641444 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641519 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641476 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641519 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641503 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641630 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641588 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641739 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641682 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token\") pod \"74c3ef72-9567-4843-983e-6b72f95ab14a\" (UID: \"74c3ef72-9567-4843-983e-6b72f95ab14a\") " Oct 30 22:03:39.641862 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.641824 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:39.642290 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.642264 5039 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-ca-configmap\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.643624 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.643598 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:39.643725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.643693 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "kube-api-access-ng88s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:39.643847 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.643829 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:39.644085 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.644069 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:39.644146 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.644124 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:39.644278 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.644253 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token" pod "74c3ef72-9567-4843-983e-6b72f95ab14a" (UID: "74c3ef72-9567-4843-983e-6b72f95ab14a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:39.743845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.743789 5039 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c3ef72-9567-4843-983e-6b72f95ab14a-local-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.743845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.743817 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.743845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.743826 5039 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-csr-dns-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.743845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.743835 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng88s\" (UniqueName: \"kubernetes.io/projected/74c3ef72-9567-4843-983e-6b72f95ab14a-kube-api-access-ng88s\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.743845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.743843 5039 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-istio-kubeconfig\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.744056 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.743852 5039 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74c3ef72-9567-4843-983e-6b72f95ab14a-cacerts\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:03:39.782721 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.782695 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.947724 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:39.947687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.101320 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.101236 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.291463 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.272446 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.470414 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.466501 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" event={"ID":"74c3ef72-9567-4843-983e-6b72f95ab14a","Type":"ContainerDied","Data":"25a0dbe44fbbf5de369500e503dc279c56f9ca73d52124596de241b9237fa565"} Oct 30 22:03:40.470414 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.466574 5039 scope.go:122] "RemoveContainer" containerID="d2badeb7112f40bc0b3398333d2649420f3a1f64d380d9efb7a5407b939ae672" Oct 30 22:03:40.470414 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.466742 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d" Oct 30 22:03:40.580541 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.580500 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d"] Oct 30 22:03:40.588642 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.588615 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4vc5d"] Oct 30 22:03:40.923346 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:40.923317 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c3ef72-9567-4843-983e-6b72f95ab14a" path="/var/lib/kubelet/pods/74c3ef72-9567-4843-983e-6b72f95ab14a/volumes" Oct 30 22:03:41.856820 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:41.856780 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:43.489429 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:43.489368 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:44.181232 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:44.181196 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:44.343094 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:44.343050 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.607807 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:45.607764 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:46.148762 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:46.148731 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:46.330662 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:46.330483 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:46.593773 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:46.593680 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:47.969600 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:47.969561 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.212732 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.212698 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.379865 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.379774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.447693 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.447656 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk"] Oct 30 22:03:48.448983 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.448948 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:03:48.449151 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449137 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:03:48.449272 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449261 5039 cpu_manager.go:414] "containerMap: removing container" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:03:48.449352 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449342 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:03:48.449468 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449456 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:03:48.449556 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449545 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:03:48.449662 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449650 5039 cpu_manager.go:414] "containerMap: removing container" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.449742 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449731 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.449890 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449879 5039 cpu_manager.go:414] "containerMap: removing container" podUID="74c3ef72-9567-4843-983e-6b72f95ab14a" containerName="discovery" Oct 30 22:03:48.449977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.449965 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c3ef72-9567-4843-983e-6b72f95ab14a" containerName="discovery" Oct 30 22:03:48.450082 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450057 5039 cpu_manager.go:414] "containerMap: removing container" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:03:48.450174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450161 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:03:48.450291 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450280 5039 cpu_manager.go:414] "containerMap: removing container" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.450368 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450358 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.450489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450477 5039 cpu_manager.go:414] "containerMap: removing container" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.450567 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450557 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.450690 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450679 5039 cpu_manager.go:414] "containerMap: removing container" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" Oct 30 22:03:48.450778 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450768 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" Oct 30 22:03:48.451000 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.450985 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:03:48.451522 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.451092 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.451667 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.451652 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:03:48.451766 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.451755 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="57417a1b-d284-4aac-9a1d-f7843068d67e" containerName="istio-proxy" Oct 30 22:03:48.451861 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.451851 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:03:48.451950 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.451941 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:03:48.452030 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.452021 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:03:48.452127 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.452116 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c3ef72-9567-4843-983e-6b72f95ab14a" containerName="discovery" Oct 30 22:03:48.457953 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.457929 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.473406 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.473217 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk"] Oct 30 22:03:48.477116 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.477095 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a42c31-openshift-default-dockercfg-k82br\"" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.497349 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.497442 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsh5s\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498363 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498411 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498476 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498548 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498577 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498735 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.499039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.498768 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.547466 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.547428 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609257 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609308 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609408 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609490 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsh5s\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609543 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609573 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609641 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609737 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.609770 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.610277 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.610600 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.613460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.610835 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.671331 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.615807 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.671590 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.657736 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.671707 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.658062 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.671787 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.663917 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.671873 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.665888 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.720038 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.714963 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsh5s\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s\") pod \"e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:48.843884 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:48.843849 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:49.027680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.027585 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:49.203525 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.203496 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk"] Oct 30 22:03:49.204253 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:03:49.204228 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff95999_64cd_49b1_aa4a_cf11bb3eb20c.slice/crio-233c6996c394da20fce4535066077654710aad9ab1fa6b09ee7d1b0973c1b8cd WatchSource:0}: Error finding container 233c6996c394da20fce4535066077654710aad9ab1fa6b09ee7d1b0973c1b8cd: Status 404 returned error can't find the container with id 233c6996c394da20fce4535066077654710aad9ab1fa6b09ee7d1b0973c1b8cd Oct 30 22:03:49.207262 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.207219 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:49.207369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.207302 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:49.207369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.207348 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:03:49.724132 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.724097 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" event={"ID":"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c","Type":"ContainerStarted","Data":"13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150"} Oct 30 22:03:49.724303 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.724156 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" event={"ID":"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c","Type":"ContainerStarted","Data":"233c6996c394da20fce4535066077654710aad9ab1fa6b09ee7d1b0973c1b8cd"} Oct 30 22:03:49.783113 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.783052 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podStartSLOduration=1.783034648 podStartE2EDuration="1.783034648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:49.777921073 +0000 UTC m=+3268.987860712" watchObservedRunningTime="2026-10-30 22:03:49.783034648 +0000 UTC m=+3268.992974277" Oct 30 22:03:49.796729 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.796695 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:49.844653 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.844625 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:49.967012 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:49.966977 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.131220 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:50.131129 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.315612 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:50.299638 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.465599 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:50.465564 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.851136 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:50.851060 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:50.954222 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:50.954185 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:51.108610 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:51.108522 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:51.738213 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:51.738181 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:51.739093 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:51.739070 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:03:52.474370 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:52.474317 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:52.626415 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:52.623984 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:52.969138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:52.969107 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:53.287334 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:53.287249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:53.458069 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:53.458031 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:53.986996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:53.986967 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.203764 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:54.203730 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.375459 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:54.375347 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.529316 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:54.529285 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.779096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:54.779057 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.184874 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:03:55.184841 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:03:55.203870 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:55.203815 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.412420 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:55.412373 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.575551 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:55.575464 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:56.565559 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:56.565528 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:57.599090 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:57.599043 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:57.760134 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:57.760097 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:58.692031 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:58.692000 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:58.841347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:58.841311 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:59.579991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:03:59.579962 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:01.287579 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:01.287541 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:02.858772 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:02.858692 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:03.020128 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:03.020092 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:03.467918 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:03.467877 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:04.405853 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:04.405814 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:04.643635 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:04.643604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:05.360971 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:05.360939 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:06.406582 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:06.406504 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:08.087720 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:08.087685 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:09.348169 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348137 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg"] Oct 30 22:04:09.348791 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348775 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:04:09.348848 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348791 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:04:09.348848 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348810 5039 cpu_manager.go:414] "containerMap: removing container" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:04:09.348848 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348815 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:04:09.348970 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348920 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3852928-e949-4388-b4a6-950f9cf70f21" containerName="cleanup" Oct 30 22:04:09.348970 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348932 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c5621-c54f-4c04-91f0-06655af72601" containerName="cleanup" Oct 30 22:04:09.348970 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.348942 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a29e410-fd4c-4016-aa8b-3e7b678b1967" containerName="cleanup" Oct 30 22:04:09.352670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.352651 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.373134 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.373111 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:04:09.373134 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.373128 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:04:09.382621 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.382604 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:04:09.382725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.382651 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l5v2k\"" Oct 30 22:04:09.388344 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.388322 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg"] Oct 30 22:04:09.404577 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.404555 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvqp\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.404673 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.404584 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.404673 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.404611 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.404797 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.404709 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.404910 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.404884 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.405125 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.405013 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.405125 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.405060 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.505971 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.505951 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506065 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506001 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506111 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506063 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506111 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506097 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506168 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvqp\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506191 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506254 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506226 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.506858 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.506834 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.508404 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.508373 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.508480 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.508385 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.508614 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.508596 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.508655 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.508623 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.523979 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.523958 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvqp\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.524121 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.524101 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-srlwg\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.665849 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.665824 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:09.782328 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.782292 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:09.974971 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.974941 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg"] Oct 30 22:04:09.976131 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:04:09.976103 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182ef68a_10a3_4119_a059_3883a088fe64.slice/crio-69e4e731ff2f35a14931848c8603d5bd563493242359abc2ff1e4ef0725d758f WatchSource:0}: Error finding container 69e4e731ff2f35a14931848c8603d5bd563493242359abc2ff1e4ef0725d758f: Status 404 returned error can't find the container with id 69e4e731ff2f35a14931848c8603d5bd563493242359abc2ff1e4ef0725d758f Oct 30 22:04:09.979048 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.979014 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:09.979128 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:09.979077 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:10.260827 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:10.260747 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:10.411758 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:10.411722 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:10.909822 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:10.909774 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" event={"ID":"182ef68a-10a3-4119-a059-3883a088fe64","Type":"ContainerStarted","Data":"868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3"} Oct 30 22:04:10.910014 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:10.909856 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" event={"ID":"182ef68a-10a3-4119-a059-3883a088fe64","Type":"ContainerStarted","Data":"69e4e731ff2f35a14931848c8603d5bd563493242359abc2ff1e4ef0725d758f"} Oct 30 22:04:10.910014 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:10.909899 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:10.967921 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:10.967858 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" podStartSLOduration=1.967846339 podStartE2EDuration="1.967846339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:10.955415285 +0000 UTC m=+3290.165354913" watchObservedRunningTime="2026-10-30 22:04:10.967846339 +0000 UTC m=+3290.177785969" Oct 30 22:04:11.756420 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:11.756366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:11.917506 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:11.917473 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:04:13.607310 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:13.607251 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:15.334991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:15.334920 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:16.263233 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.263202 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq"] Oct 30 22:04:16.267353 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.267333 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.294808 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.294784 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea40ee-openshift-default-dockercfg-8mf6h\"" Oct 30 22:04:16.300174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.300147 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq"] Oct 30 22:04:16.399316 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399288 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399316 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399318 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399353 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399489 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399547 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399594 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399652 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399687 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjf5l\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.399918 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.399726 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.500831 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.500804 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.500935 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.500851 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.500935 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.500889 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjf5l\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.500939 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.500969 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.500993 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501163 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501042 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501163 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501123 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501224 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501417 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501369 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501494 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501424 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501553 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501503 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501606 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501582 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.501661 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.501644 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.503346 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.503327 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.503601 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.503584 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.517697 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.517641 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.517796 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.517772 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjf5l\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.583368 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.583347 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:16.895255 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.895231 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq"] Oct 30 22:04:16.895745 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:04:16.895725 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba35083_ab20_4a9d_a265_1238b03c7d07.slice/crio-8338ef937cc3cf3c4150870e5d61f2b5a6d60105a7ca7b377f124485a1042322 WatchSource:0}: Error finding container 8338ef937cc3cf3c4150870e5d61f2b5a6d60105a7ca7b377f124485a1042322: Status 404 returned error can't find the container with id 8338ef937cc3cf3c4150870e5d61f2b5a6d60105a7ca7b377f124485a1042322 Oct 30 22:04:16.902201 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.902164 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:16.902318 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.902234 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:16.902318 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.902264 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:04:16.950029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.950005 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" event={"ID":"eba35083-ab20-4a9d-a265-1238b03c7d07","Type":"ContainerStarted","Data":"8338ef937cc3cf3c4150870e5d61f2b5a6d60105a7ca7b377f124485a1042322"} Oct 30 22:04:16.974281 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:16.974241 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:17.124170 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:17.124131 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:17.981829 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:17.981787 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" event={"ID":"eba35083-ab20-4a9d-a265-1238b03c7d07","Type":"ContainerStarted","Data":"7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97"} Oct 30 22:04:18.034195 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:18.034149 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podStartSLOduration=2.034135237 podStartE2EDuration="2.034135237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:18.031556343 +0000 UTC m=+3297.241495973" watchObservedRunningTime="2026-10-30 22:04:18.034135237 +0000 UTC m=+3297.244075180" Oct 30 22:04:18.584157 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:18.583737 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:18.585028 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:18.584999 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.585277 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:18.585249 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:04:19.583749 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:19.583717 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.584136 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:19.583766 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:04:19.854835 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:19.854739 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:20.583809 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:20.583738 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:20.583809 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:20.583794 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:04:20.734771 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:20.734738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:20.947051 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:20.947020 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:04:21.203782 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:21.203706 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:21.356835 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:21.356804 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:21.424141 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:21.424104 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:04:21.442181 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:21.442159 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:04:21.583854 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:21.583763 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:21.583854 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:21.583839 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:04:22.583754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:22.583723 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:22.583977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:22.583766 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:04:23.584258 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:23.584214 5039 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:23.584766 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:23.584281 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 22:04:24.597410 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:24.596933 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:24.612407 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:24.598714 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:24.612407 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:24.603677 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:04:25.001216 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:25.001177 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:25.252706 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:04:25.252565 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:04:31.942090 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:31.942051 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:32.216580 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:32.216498 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:33.937869 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:33.937785 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:35.772790 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:35.772751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:37.456152 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:37.456117 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:37.598797 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:37.598763 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:39.190873 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:39.190840 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:40.729868 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:40.729832 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:42.315976 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:42.315925 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:43.938349 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:43.938312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:44.083359 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:44.083324 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:44.226202 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:44.226128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:45.715124 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:45.715087 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:47.499716 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:47.499678 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:49.055511 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:49.055457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:50.702174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:50.702139 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:52.153490 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:52.153452 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:52.651996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:52.651957 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:52.863628 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:52.863587 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:53.034474 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:53.034323 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:53.365084 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:53.364995 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:54.801832 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:54.801798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:55.184800 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:04:55.184761 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:04:56.359308 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:56.359271 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:57.911800 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:57.911704 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.371268 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:59.371227 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.541377 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:59.541339 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.657220 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:59.657184 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w"] Oct 30 22:04:59.657895 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:59.657851 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" containerID="cri-o://2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51" gracePeriod=30 Oct 30 22:04:59.695639 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:59.695607 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.858956 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:04:59.858766 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:01.565886 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:01.565851 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:03.157215 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:03.157174 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:04.768854 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:04.768805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:04.944845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:04.944791 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 22:05:05.047546 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047515 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047582 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047768 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047864 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047815 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047864 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047846 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047886 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047920 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.047977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.047961 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.048130 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.048101 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket\") pod \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\" (UID: \"8e574d24-b86a-48cd-af75-0bd5651d0ca2\") " Oct 30 22:05:05.048130 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.048104 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:05.048511 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.048490 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:05.048794 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.048747 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:05.048869 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.048811 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:05.049358 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.049334 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:05.049552 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.049534 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.049666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.049648 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.049757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.049671 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.049757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.049684 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.051558 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.050190 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:05.051558 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.050584 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:05.051865 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.051845 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:05.051913 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.051890 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj" pod "8e574d24-b86a-48cd-af75-0bd5651d0ca2" (UID: "8e574d24-b86a-48cd-af75-0bd5651d0ca2"). InnerVolumeSpecName "kube-api-access-7zktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:05.151454 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.151428 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.151454 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.151452 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.151600 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.151460 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.151600 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.151468 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e574d24-b86a-48cd-af75-0bd5651d0ca2-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.151600 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.151476 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zktj\" (UniqueName: \"kubernetes.io/projected/8e574d24-b86a-48cd-af75-0bd5651d0ca2-kube-api-access-7zktj\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:05:05.432204 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.432173 5039 generic.go:356] "Generic (PLEG): container finished" podID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerID="2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51" exitCode=0 Oct 30 22:05:05.432327 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.432223 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" Oct 30 22:05:05.432327 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.432219 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" event={"ID":"8e574d24-b86a-48cd-af75-0bd5651d0ca2","Type":"ContainerDied","Data":"2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51"} Oct 30 22:05:05.432460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.432353 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w" event={"ID":"8e574d24-b86a-48cd-af75-0bd5651d0ca2","Type":"ContainerDied","Data":"b2eaf31132eb38310129edbcfa36b48903a093170e9cc5a007f204db15974038"} Oct 30 22:05:05.432460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.432383 5039 scope.go:122] "RemoveContainer" containerID="2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51" Oct 30 22:05:05.442830 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.442815 5039 scope.go:122] "RemoveContainer" containerID="91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617" Oct 30 22:05:05.451752 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.451737 5039 scope.go:122] "RemoveContainer" containerID="2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51" Oct 30 22:05:05.451983 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:05:05.451966 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51\": container with ID starting with 2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51 not found: ID does not exist" containerID="2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51" Oct 30 22:05:05.452031 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.451990 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51"} err="failed to get container status \"2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51\": rpc error: code = NotFound desc = could not find container \"2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51\": container with ID starting with 2dbc9666f5b1d8c753f2669d852853213a2a33c2cb6b4815f3736da0c3f8bc51 not found: ID does not exist" Oct 30 22:05:05.452031 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.452015 5039 scope.go:122] "RemoveContainer" containerID="91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617" Oct 30 22:05:05.452239 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:05:05.452224 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617\": container with ID starting with 91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617 not found: ID does not exist" containerID="91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617" Oct 30 22:05:05.452279 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.452247 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617"} err="failed to get container status \"91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617\": rpc error: code = NotFound desc = could not find container \"91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617\": container with ID starting with 91e297ae8ecd79116ae3cb386521444bf3a27201422a9f50d16aec926664e617 not found: ID does not exist" Oct 30 22:05:05.496223 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.496204 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w"] Oct 30 22:05:05.505600 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:05.505578 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-n8k5w"] Oct 30 22:05:06.510147 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:06.510108 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:06.922735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:06.922708 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" path="/var/lib/kubelet/pods/8e574d24-b86a-48cd-af75-0bd5651d0ca2/volumes" Oct 30 22:05:08.225159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:08.225117 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:09.912233 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:09.909067 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:11.004024 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:11.003992 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:11.160615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:11.160579 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:11.840963 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:11.840917 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:13.398733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:13.398694 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.026883 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:15.026815 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.330777 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:15.328774 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:05:16.825552 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:16.825510 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:16.985062 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:16.985021 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:17.180412 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:17.179669 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:18.784577 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:18.784540 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.410368 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:19.410336 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.556415 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:19.555818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:20.239882 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:20.239841 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:21.212113 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:21.212078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:21.427354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:21.427318 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:21.584026 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:21.583931 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:21.776234 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:21.776191 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:21.946034 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:21.945762 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.240594 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:22.240504 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.395416 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:22.395366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:23.481727 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:23.481688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:24.139870 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:24.139821 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:25.184892 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:05:25.184858 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:05:25.429006 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.428966 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:25.611783 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.611756 5039 generic.go:356] "Generic (PLEG): container finished" podID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerID="13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150" exitCode=0 Oct 30 22:05:25.611922 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.611831 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" event={"ID":"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c","Type":"ContainerDied","Data":"13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150"} Oct 30 22:05:25.612278 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.612262 5039 scope.go:122] "RemoveContainer" containerID="13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150" Oct 30 22:05:25.614070 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.614039 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:25.614143 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.614114 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:25.614178 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:25.614146 5039 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:26.370372 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:26.370336 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:26.621366 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:26.621285 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" event={"ID":"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c","Type":"ContainerStarted","Data":"850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8"} Oct 30 22:05:26.840804 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:26.840772 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:26.844561 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:26.844545 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:05:26.844732 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:26.844714 5039 patch_prober.go:29] interesting pod/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:05:26.844776 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:26.844756 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:05:27.266412 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:27.263012 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.464412 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:27.464172 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.670853 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:27.670741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.845009 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:27.844973 5039 patch_prober.go:29] interesting pod/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.845187 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:27.845044 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:05:27.850250 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:27.850219 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:28.844963 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:28.844933 5039 patch_prober.go:29] interesting pod/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.845330 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:28.844980 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:05:28.994851 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:28.994816 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.249640 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:29.249602 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.418723 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:29.418682 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.783368 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:29.783334 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.845108 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:29.845081 5039 patch_prober.go:29] interesting pod/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:05:29.845483 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:29.845123 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:05:30.524589 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:30.524548 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:30.836712 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:30.836615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:30.849178 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:30.849135 5039 patch_prober.go:29] interesting pod/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 22:05:30.850898 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:30.850861 5039 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 22:05:31.190176 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:31.190140 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.387414 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:31.386872 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.557332 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:31.557254 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.934363 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:31.934335 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:05:31.934863 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:31.934602 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:05:31.935216 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:31.935197 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:05:33.214412 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:33.214325 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:33.416963 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:33.416928 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:34.696574 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:34.696544 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:34.895415 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:34.888457 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:35.128810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:35.128724 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:35.681755 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:35.681713 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:35.826505 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:35.826465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:36.000489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:35.999238 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:36.608965 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:36.608930 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:36.793934 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:36.793897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:38.062302 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:38.062267 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:39.846929 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:39.846897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:40.025458 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:40.025422 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:41.621997 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:41.621963 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:41.841448 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:41.831854 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:42.780184 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:42.780149 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:43.524810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:43.524769 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:45.196712 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:45.196681 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:46.550515 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:46.550472 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.224081 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:47.224049 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.384350 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:47.384322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.528051 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:47.527959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.831781 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:47.831701 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:49.253243 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:49.253213 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:51.576674 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:51.576639 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:53.358652 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:53.355976 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:54.268522 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:54.268485 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:54.504451 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:54.504380 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:54.679754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:54.679722 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:55.184484 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:05:55.184447 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:05:55.395424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:55.395374 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:57.205263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:57.205224 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:58.928891 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:05:58.928850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:00.705286 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:00.705249 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:02.360596 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:02.360556 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:04.020970 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:04.020920 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:05.624459 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:05.621967 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:05.879667 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:05.879565 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:06.029571 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:06.029534 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:07.295694 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:07.295658 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:08.602529 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:08.602490 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66"] Oct 30 22:06:08.604011 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:08.603972 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" podUID="d8473744-30a3-4ea3-8e50-9fd5ad981ff0" containerName="maas-api" containerID="cri-o://f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19" gracePeriod=30 Oct 30 22:06:08.710005 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:08.709964 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:09.117581 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.117549 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 22:06:09.129505 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.127563 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca\") pod \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " Oct 30 22:06:09.129505 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.127626 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57\" (UniqueName: \"kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57\") pod \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " Oct 30 22:06:09.129505 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.127822 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\") pod \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " Oct 30 22:06:09.129505 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.127955 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs\") pod \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\" (UID: \"d8473744-30a3-4ea3-8e50-9fd5ad981ff0\") " Oct 30 22:06:09.131433 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.131384 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca" pod "d8473744-30a3-4ea3-8e50-9fd5ad981ff0" (UID: "d8473744-30a3-4ea3-8e50-9fd5ad981ff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:09.133586 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.133548 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs" pod "d8473744-30a3-4ea3-8e50-9fd5ad981ff0" (UID: "d8473744-30a3-4ea3-8e50-9fd5ad981ff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:09.138260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.137746 5039 generic.go:356] "Generic (PLEG): container finished" podID="d8473744-30a3-4ea3-8e50-9fd5ad981ff0" containerID="f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19" exitCode=0 Oct 30 22:06:09.138260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.137932 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" event={"ID":"d8473744-30a3-4ea3-8e50-9fd5ad981ff0","Type":"ContainerDied","Data":"f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19"} Oct 30 22:06:09.138260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.137970 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" event={"ID":"d8473744-30a3-4ea3-8e50-9fd5ad981ff0","Type":"ContainerDied","Data":"cd4c9bb8431ce4ea087a98c2c33290f078709c43ec8d51ef251f66a34ba5c7fe"} Oct 30 22:06:09.138260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.137988 5039 scope.go:122] "RemoveContainer" containerID="f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19" Oct 30 22:06:09.138260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.138099 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66" Oct 30 22:06:09.140474 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.140287 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls" pod "d8473744-30a3-4ea3-8e50-9fd5ad981ff0" (UID: "d8473744-30a3-4ea3-8e50-9fd5ad981ff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:09.140474 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.140353 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57" pod "d8473744-30a3-4ea3-8e50-9fd5ad981ff0" (UID: "d8473744-30a3-4ea3-8e50-9fd5ad981ff0"). InnerVolumeSpecName "kube-api-access-l6h57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:09.170904 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.170796 5039 scope.go:122] "RemoveContainer" containerID="f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19" Oct 30 22:06:09.171205 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:06:09.171173 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19\": container with ID starting with f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19 not found: ID does not exist" containerID="f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19" Oct 30 22:06:09.171317 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.171216 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19"} err="failed to get container status \"f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19\": rpc error: code = NotFound desc = could not find container \"f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19\": container with ID starting with f522d09e80033752020ec4f1a3c0ec6d804bb54a09795279ab5913a99651fc19 not found: ID does not exist" Oct 30 22:06:09.229530 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.229493 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:09.229530 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.229526 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6h57\" (UniqueName: \"kubernetes.io/projected/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-kube-api-access-l6h57\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:09.229530 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.229537 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:09.229684 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.229546 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8473744-30a3-4ea3-8e50-9fd5ad981ff0-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:09.444524 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.444493 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:09.482893 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.482868 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66"] Oct 30 22:06:09.493246 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.493224 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3172e5-565d8cd46f-z2j66"] Oct 30 22:06:09.609980 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:09.609939 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:10.040894 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:10.040857 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:10.922695 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:10.922662 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8473744-30a3-4ea3-8e50-9fd5ad981ff0" path="/var/lib/kubelet/pods/d8473744-30a3-4ea3-8e50-9fd5ad981ff0/volumes" Oct 30 22:06:11.314641 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:11.314541 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:12.877893 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:12.877807 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:14.464299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:14.464266 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:14.609497 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:14.609464 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:16.189430 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:16.189366 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:16.347834 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:16.347798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:16.502798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:16.502716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:16.679413 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:16.673464 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:17.250252 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:17.250221 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:18.253224 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:18.253195 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:18.407903 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:18.407866 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:18.652979 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:18.652946 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:20.253043 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:20.253012 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:21.993915 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:21.993880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:22.285640 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:22.285567 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:06:23.674936 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:23.674900 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:25.184603 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:06:25.184570 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:06:25.191995 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.191972 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb"] Oct 30 22:06:25.192628 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192612 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d8473744-30a3-4ea3-8e50-9fd5ad981ff0" containerName="maas-api" Oct 30 22:06:25.192681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192632 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8473744-30a3-4ea3-8e50-9fd5ad981ff0" containerName="maas-api" Oct 30 22:06:25.192714 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192682 5039 cpu_manager.go:414] "containerMap: removing container" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" Oct 30 22:06:25.192714 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192689 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" Oct 30 22:06:25.192714 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192707 5039 cpu_manager.go:414] "containerMap: removing container" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" Oct 30 22:06:25.192808 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192715 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" Oct 30 22:06:25.192879 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192868 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" Oct 30 22:06:25.192913 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192886 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e574d24-b86a-48cd-af75-0bd5651d0ca2" containerName="istio-proxy" Oct 30 22:06:25.192913 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.192902 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8473744-30a3-4ea3-8e50-9fd5ad981ff0" containerName="maas-api" Oct 30 22:06:25.196677 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.196651 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.209681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.209657 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-19a725-serving-cert\"" Oct 30 22:06:25.214374 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.214351 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb"] Oct 30 22:06:25.326076 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.326045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:25.333492 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.333468 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.333621 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.333525 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.333621 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.333610 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.333702 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.333636 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvv4\" (UniqueName: \"kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.435909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.435824 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.435909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.435893 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.436174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.435925 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.436174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.435946 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvv4\" (UniqueName: \"kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.436998 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.436967 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.438915 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.438888 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.439146 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.439102 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.453965 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.453937 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvv4\" (UniqueName: \"kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4\") pod \"maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.482874 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.482832 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:25.510634 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.510608 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:25.824680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.824653 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb"] Oct 30 22:06:25.826566 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:06:25.826537 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb181f48_bb52_4eff_8a23_d792758d7f53.slice/crio-1e8d1ed26556c74561b32cf505a358b4bda9042185c717fa93fb19df8855d7c0 WatchSource:0}: Error finding container 1e8d1ed26556c74561b32cf505a358b4bda9042185c717fa93fb19df8855d7c0: Status 404 returned error can't find the container with id 1e8d1ed26556c74561b32cf505a358b4bda9042185c717fa93fb19df8855d7c0 Oct 30 22:06:25.827892 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:25.827873 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:06:26.257102 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:26.257072 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" event={"ID":"cb181f48-bb52-4eff-8a23-d792758d7f53","Type":"ContainerStarted","Data":"1e8d1ed26556c74561b32cf505a358b4bda9042185c717fa93fb19df8855d7c0"} Oct 30 22:06:27.020495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:27.019947 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:27.266040 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:27.266007 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" event={"ID":"cb181f48-bb52-4eff-8a23-d792758d7f53","Type":"ContainerStarted","Data":"5c426cb2b1d309be5a89f759ef4ca2c656cf40e76cd7d884128a4f538ee60ff0"} Oct 30 22:06:27.266472 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:27.266070 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:27.302931 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:27.302833 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" podStartSLOduration=1.8813502899999999 podStartE2EDuration="2.30281633s" totalImagesPullingTime="421.46604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:27.301126564 +0000 UTC m=+3426.511066175" watchObservedRunningTime="2026-10-30 22:06:27.30281633 +0000 UTC m=+3426.512755960" Oct 30 22:06:27.749179 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:27.749141 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:27.908370 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:27.908331 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:28.809646 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:28.809610 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:30.420352 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:30.420248 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:31.955044 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:31.955009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:32.615289 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:32.615255 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:32.783531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:32.783502 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:32.936772 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:32.936741 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.263685 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:33.263597 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.309038 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:33.309016 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:06:34.641441 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:34.641380 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:36.204011 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:36.203978 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:37.880725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:37.877775 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.506749 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:39.506715 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.657285 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:39.657245 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.853199 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:39.853110 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2"] Oct 30 22:06:39.853491 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:39.853454 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" containerID="cri-o://52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39" gracePeriod=30 Oct 30 22:06:39.935256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:39.935224 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:40.358530 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:40.358494 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:41.364972 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:41.364932 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:43.121554 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:43.121514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:44.774801 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:44.774769 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.133168 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.133137 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 22:06:45.264437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264402 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264438 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264682 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264483 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264682 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264571 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264682 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264599 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264683 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264726 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264777 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.264842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264811 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs\") pod \"7ac72929-ed78-471f-8d8d-49f5dde68520\" (UID: \"7ac72929-ed78-471f-8d8d-49f5dde68520\") " Oct 30 22:06:45.265055 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.264845 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:45.265115 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265050 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:45.265271 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265248 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:45.265355 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265326 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:45.265434 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265179 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:45.265498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265449 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.265498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265471 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.265498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.265487 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.266647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.266626 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "kube-api-access-ztzcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:45.266887 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.266870 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:45.267094 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.267068 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:45.267170 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.267139 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo" pod "7ac72929-ed78-471f-8d8d-49f5dde68520" (UID: "7ac72929-ed78-471f-8d8d-49f5dde68520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:06:45.276329 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.276305 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg"] Oct 30 22:06:45.366906 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.366852 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac72929-ed78-471f-8d8d-49f5dde68520-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.366906 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.366875 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.366906 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.366883 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.366906 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.366895 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzcz\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-kube-api-access-ztzcz\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.366906 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.366904 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.367137 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.366912 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac72929-ed78-471f-8d8d-49f5dde68520-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:06:45.501689 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.501654 5039 generic.go:356] "Generic (PLEG): container finished" podID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerID="52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39" exitCode=0 Oct 30 22:06:45.501842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.501705 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" Oct 30 22:06:45.501842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.501711 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" event={"ID":"7ac72929-ed78-471f-8d8d-49f5dde68520","Type":"ContainerDied","Data":"52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39"} Oct 30 22:06:45.501842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.501750 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2" event={"ID":"7ac72929-ed78-471f-8d8d-49f5dde68520","Type":"ContainerDied","Data":"c4fffeaba9e38ea3ab6449ae9a66c98a0504d2928fb0e53e616f23c486dba280"} Oct 30 22:06:45.501842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.501770 5039 scope.go:122] "RemoveContainer" containerID="52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39" Oct 30 22:06:45.512495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.512437 5039 scope.go:122] "RemoveContainer" containerID="3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8" Oct 30 22:06:45.516459 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.516379 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.522762 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.522743 5039 scope.go:122] "RemoveContainer" containerID="52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39" Oct 30 22:06:45.522996 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:06:45.522976 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39\": container with ID starting with 52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39 not found: ID does not exist" containerID="52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39" Oct 30 22:06:45.523076 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.523008 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39"} err="failed to get container status \"52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39\": rpc error: code = NotFound desc = could not find container \"52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39\": container with ID starting with 52cbcce40f26052d694e1512eff51f1cb778e115d14c2e773aa577bc1c114b39 not found: ID does not exist" Oct 30 22:06:45.523076 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.523030 5039 scope.go:122] "RemoveContainer" containerID="3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8" Oct 30 22:06:45.523282 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:06:45.523265 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8\": container with ID starting with 3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8 not found: ID does not exist" containerID="3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8" Oct 30 22:06:45.523319 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.523288 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8"} err="failed to get container status \"3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8\": rpc error: code = NotFound desc = could not find container \"3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8\": container with ID starting with 3204773e1a6cc6f4dfc4f03d97a76fb4bc1f215cb696309e0e40a430718539a8 not found: ID does not exist" Oct 30 22:06:45.548945 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.548920 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2"] Oct 30 22:06:45.553400 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:45.553367 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f49db2-openshift-default-ddc44fb5f-cxjl2"] Oct 30 22:06:46.409191 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:46.409158 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:46.921900 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:46.921869 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" path="/var/lib/kubelet/pods/7ac72929-ed78-471f-8d8d-49f5dde68520/volumes" Oct 30 22:06:48.106401 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:48.106362 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:49.764648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:49.764615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.217046 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:51.217003 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.354935 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:51.354899 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.504060 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:51.503987 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:53.112420 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:53.112362 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:53.270000 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:53.269973 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:06:54.988812 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:54.988780 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:55.185377 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:06:55.185339 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:06:56.725137 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:56.725103 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:58.383880 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:06:58.383843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:00.257893 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.257847 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:00.296845 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.296804 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" containerID="cri-o://48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220" gracePeriod=45 Oct 30 22:07:00.564117 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.564089 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 22:07:00.612553 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.612520 5039 generic.go:356] "Generic (PLEG): container finished" podID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerID="48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220" exitCode=0 Oct 30 22:07:00.612708 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.612627 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" event={"ID":"66188eb2-2660-4cca-9fcf-e70428ee5286","Type":"ContainerDied","Data":"48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220"} Oct 30 22:07:00.612708 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.612674 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" event={"ID":"66188eb2-2660-4cca-9fcf-e70428ee5286","Type":"ContainerDied","Data":"454424fd6c86a8994fd904acd346d2ef0d7217aaee4371f64310640a5d2ee792"} Oct 30 22:07:00.612708 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.612696 5039 scope.go:122] "RemoveContainer" containerID="48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220" Oct 30 22:07:00.612881 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.612632 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg" Oct 30 22:07:00.625023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.625002 5039 scope.go:122] "RemoveContainer" containerID="48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220" Oct 30 22:07:00.625356 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:07:00.625306 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220\": container with ID starting with 48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220 not found: ID does not exist" containerID="48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220" Oct 30 22:07:00.625479 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.625349 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220"} err="failed to get container status \"48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220\": rpc error: code = NotFound desc = could not find container \"48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220\": container with ID starting with 48145698156d7a191f523f73c5440af9ffade0c222917f872008e70d1ee10220 not found: ID does not exist" Oct 30 22:07:00.741957 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.741928 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs\" (UniqueName: \"kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs\") pod \"66188eb2-2660-4cca-9fcf-e70428ee5286\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " Oct 30 22:07:00.742139 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.741971 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home\") pod \"66188eb2-2660-4cca-9fcf-e70428ee5286\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " Oct 30 22:07:00.742139 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742043 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn\" (UniqueName: \"kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn\") pod \"66188eb2-2660-4cca-9fcf-e70428ee5286\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " Oct 30 22:07:00.742139 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742085 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir\") pod \"66188eb2-2660-4cca-9fcf-e70428ee5286\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " Oct 30 22:07:00.742336 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742163 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache\") pod \"66188eb2-2660-4cca-9fcf-e70428ee5286\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " Oct 30 22:07:00.742336 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742251 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home" pod "66188eb2-2660-4cca-9fcf-e70428ee5286" (UID: "66188eb2-2660-4cca-9fcf-e70428ee5286"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:00.742336 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742257 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm\") pod \"66188eb2-2660-4cca-9fcf-e70428ee5286\" (UID: \"66188eb2-2660-4cca-9fcf-e70428ee5286\") " Oct 30 22:07:00.742504 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742448 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir" pod "66188eb2-2660-4cca-9fcf-e70428ee5286" (UID: "66188eb2-2660-4cca-9fcf-e70428ee5286"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:00.742504 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742470 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache" pod "66188eb2-2660-4cca-9fcf-e70428ee5286" (UID: "66188eb2-2660-4cca-9fcf-e70428ee5286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:00.742867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742845 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:07:00.742867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742864 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:07:00.743033 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.742874 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:07:00.744186 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.744167 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs" pod "66188eb2-2660-4cca-9fcf-e70428ee5286" (UID: "66188eb2-2660-4cca-9fcf-e70428ee5286"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:00.744276 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.744215 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm" pod "66188eb2-2660-4cca-9fcf-e70428ee5286" (UID: "66188eb2-2660-4cca-9fcf-e70428ee5286"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:00.744316 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.744273 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn" pod "66188eb2-2660-4cca-9fcf-e70428ee5286" (UID: "66188eb2-2660-4cca-9fcf-e70428ee5286"). InnerVolumeSpecName "kube-api-access-jrzmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:00.844060 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.843993 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzmn\" (UniqueName: \"kubernetes.io/projected/66188eb2-2660-4cca-9fcf-e70428ee5286-kube-api-access-jrzmn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:07:00.844060 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.844019 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66188eb2-2660-4cca-9fcf-e70428ee5286-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:07:00.844060 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.844029 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66188eb2-2660-4cca-9fcf-e70428ee5286-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:07:00.955609 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.955579 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg"] Oct 30 22:07:00.965188 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:00.965159 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-2-w0-49738b-kserve-fd48fd8b9-dfwmg"] Oct 30 22:07:02.392167 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:02.392123 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:02.922189 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:02.922156 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" path="/var/lib/kubelet/pods/66188eb2-2660-4cca-9fcf-e70428ee5286/volumes" Oct 30 22:07:03.180938 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:03.180837 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:04.536058 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:04.536022 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:04.990263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:04.990191 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:06.484171 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:06.484111 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:06.676155 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:06.676119 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:06.903355 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:06.903323 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:07.356265 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:07.356175 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:07.531520 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:07.531480 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:08.683434 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:08.683381 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:09.589307 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:09.589277 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:10.499739 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:10.499701 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:11.001828 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:11.001795 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:11.167719 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:11.167688 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:12.536491 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:12.536452 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:13.531883 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:13.531849 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:13.705697 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:13.705650 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:14.848498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:14.848464 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:16.851796 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:16.851750 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:17.014428 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:17.014368 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:17.359553 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:17.359516 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.719946 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:18.719908 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:20.431285 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:20.429291 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.137000 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:22.136962 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:23.106053 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:23.106018 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:23.283008 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:23.282045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:24.018009 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:24.017970 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.184229 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:07:25.184190 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:07:25.676186 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:25.676146 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.970929 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:25.970841 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:27.544099 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:27.544064 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:29.067490 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:29.067441 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:29.233133 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:29.233095 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:30.830193 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:30.830160 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:30.884996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:30.884966 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:07:31.369508 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:31.369461 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.801198 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:32.801038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:34.425451 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:34.425412 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:36.114337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:36.114296 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:37.683294 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:37.683252 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:39.412549 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:39.412509 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:41.010608 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:41.010566 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:41.951325 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:41.951287 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:42.788801 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:42.786903 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:44.398907 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:44.398871 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:46.097847 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:46.096040 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:47.725598 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:47.722516 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:49.770241 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:49.770206 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:51.693432 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:51.693377 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:53.314348 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:53.314312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:53.475843 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:53.475792 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:55.121632 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:55.121596 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:55.186664 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:07:55.186631 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:07:56.804681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:56.804645 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.124754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:57.124666 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.268203 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:57.268165 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.788832 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:57.788798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:58.155227 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:58.155178 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:58.846912 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:07:58.846881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:00.487268 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:00.485906 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:02.178567 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:02.178527 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:03.794875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:03.794815 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:03.951045 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:03.951009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:05.697203 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:05.697168 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.733547 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:06.733494 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:08:07.442361 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:07.442322 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:09.225841 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:09.225801 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:10.986647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:10.986616 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:12.751880 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:12.749897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:14.509428 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:14.506195 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:14.812179 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:14.812098 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:14.972294 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:14.972259 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.939019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:15.938990 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796"] Oct 30 22:08:16.044241 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.044204 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:16.155507 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.155471 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" podUID="3ed4bc48-36e8-4858-b38e-2f3dbf766df3" containerName="maas-api" containerID="cri-o://a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5" gracePeriod=30 Oct 30 22:08:16.546401 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.546365 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 22:08:16.581964 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.581936 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca\") pod \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " Oct 30 22:08:16.582097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.582003 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls\") pod \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " Oct 30 22:08:16.582097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.582039 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs\") pod \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " Oct 30 22:08:16.582183 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.582132 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn\" (UniqueName: \"kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn\") pod \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\" (UID: \"3ed4bc48-36e8-4858-b38e-2f3dbf766df3\") " Oct 30 22:08:16.582302 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.582277 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca" pod "3ed4bc48-36e8-4858-b38e-2f3dbf766df3" (UID: "3ed4bc48-36e8-4858-b38e-2f3dbf766df3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:16.582731 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.582706 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:16.584258 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.584222 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls" pod "3ed4bc48-36e8-4858-b38e-2f3dbf766df3" (UID: "3ed4bc48-36e8-4858-b38e-2f3dbf766df3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:16.585374 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.584859 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn" pod "3ed4bc48-36e8-4858-b38e-2f3dbf766df3" (UID: "3ed4bc48-36e8-4858-b38e-2f3dbf766df3"). InnerVolumeSpecName "kube-api-access-fdlvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:16.601888 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.601849 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs" pod "3ed4bc48-36e8-4858-b38e-2f3dbf766df3" (UID: "3ed4bc48-36e8-4858-b38e-2f3dbf766df3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:16.677654 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.677627 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:16.683934 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.683912 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdlvn\" (UniqueName: \"kubernetes.io/projected/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-kube-api-access-fdlvn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:16.683934 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.683935 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:16.684054 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.683943 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed4bc48-36e8-4858-b38e-2f3dbf766df3-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:16.837087 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:16.832128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.162602 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.162570 5039 generic.go:356] "Generic (PLEG): container finished" podID="3ed4bc48-36e8-4858-b38e-2f3dbf766df3" containerID="a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5" exitCode=0 Oct 30 22:08:17.162973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.162629 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" Oct 30 22:08:17.162973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.162652 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" event={"ID":"3ed4bc48-36e8-4858-b38e-2f3dbf766df3","Type":"ContainerDied","Data":"a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5"} Oct 30 22:08:17.162973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.162716 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796" event={"ID":"3ed4bc48-36e8-4858-b38e-2f3dbf766df3","Type":"ContainerDied","Data":"2a904d0a47a2a0a6ffddafad98dbcf9f062943763d60f3dcbd4b1032f73a1340"} Oct 30 22:08:17.162973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.162745 5039 scope.go:122] "RemoveContainer" containerID="a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5" Oct 30 22:08:17.172963 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.172941 5039 scope.go:122] "RemoveContainer" containerID="a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5" Oct 30 22:08:17.173247 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:17.173228 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5\": container with ID starting with a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5 not found: ID does not exist" containerID="a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5" Oct 30 22:08:17.173308 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.173267 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5"} err="failed to get container status \"a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5\": rpc error: code = NotFound desc = could not find container \"a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5\": container with ID starting with a95a4a57138ccbbf829863a14d7629a582f4e837f2bb195114c310dcefccddd5 not found: ID does not exist" Oct 30 22:08:17.209012 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.208989 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796"] Oct 30 22:08:17.218536 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.218513 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-49738b-5544d4cc88-wk796"] Oct 30 22:08:17.284897 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.284869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.467259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:17.467193 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:18.253826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.253797 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g"] Oct 30 22:08:18.254275 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.253992 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" podUID="f3cc7de8-a788-4d88-a3f0-d14270a16b7b" containerName="maas-api" containerID="cri-o://0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604" gracePeriod=30 Oct 30 22:08:18.377449 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.377410 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:18.852482 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.852456 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 22:08:18.926099 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.926064 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed4bc48-36e8-4858-b38e-2f3dbf766df3" path="/var/lib/kubelet/pods/3ed4bc48-36e8-4858-b38e-2f3dbf766df3/volumes" Oct 30 22:08:18.939336 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.939301 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls\") pod \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " Oct 30 22:08:18.939515 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.939471 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs\") pod \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " Oct 30 22:08:18.939596 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.939551 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf\" (UniqueName: \"kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf\") pod \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " Oct 30 22:08:18.939701 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.939679 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca\") pod \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\" (UID: \"f3cc7de8-a788-4d88-a3f0-d14270a16b7b\") " Oct 30 22:08:18.940112 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.940079 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca" pod "f3cc7de8-a788-4d88-a3f0-d14270a16b7b" (UID: "f3cc7de8-a788-4d88-a3f0-d14270a16b7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:18.941864 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.941840 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls" pod "f3cc7de8-a788-4d88-a3f0-d14270a16b7b" (UID: "f3cc7de8-a788-4d88-a3f0-d14270a16b7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.942216 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.942193 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf" pod "f3cc7de8-a788-4d88-a3f0-d14270a16b7b" (UID: "f3cc7de8-a788-4d88-a3f0-d14270a16b7b"). InnerVolumeSpecName "kube-api-access-ppqzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:18.942307 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.942223 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs" pod "f3cc7de8-a788-4d88-a3f0-d14270a16b7b" (UID: "f3cc7de8-a788-4d88-a3f0-d14270a16b7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.972109 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:18.972084 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:19.041573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.041550 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppqzf\" (UniqueName: \"kubernetes.io/projected/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-kube-api-access-ppqzf\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:19.041671 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.041575 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:19.041671 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.041600 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:19.041671 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.041612 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cc7de8-a788-4d88-a3f0-d14270a16b7b-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:19.127623 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.126687 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:19.177867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.177838 5039 generic.go:356] "Generic (PLEG): container finished" podID="f3cc7de8-a788-4d88-a3f0-d14270a16b7b" containerID="0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604" exitCode=0 Oct 30 22:08:19.178035 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.177897 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" Oct 30 22:08:19.178035 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.177905 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" event={"ID":"f3cc7de8-a788-4d88-a3f0-d14270a16b7b","Type":"ContainerDied","Data":"0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604"} Oct 30 22:08:19.178035 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.177956 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g" event={"ID":"f3cc7de8-a788-4d88-a3f0-d14270a16b7b","Type":"ContainerDied","Data":"f2244cc753a2a158cc1f02ea589c996dba66a78fc876ed385a165c09ea8ab9f4"} Oct 30 22:08:19.178035 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.177982 5039 scope.go:122] "RemoveContainer" containerID="0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604" Oct 30 22:08:19.189364 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.189342 5039 scope.go:122] "RemoveContainer" containerID="0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604" Oct 30 22:08:19.189857 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:19.189830 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604\": container with ID starting with 0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604 not found: ID does not exist" containerID="0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604" Oct 30 22:08:19.189983 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.189867 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604"} err="failed to get container status \"0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604\": rpc error: code = NotFound desc = could not find container \"0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604\": container with ID starting with 0ce108e328d01a0bbab40504cc513d1f82bf746bec10a9b32d4d1cf11d4a9604 not found: ID does not exist" Oct 30 22:08:19.226431 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.226346 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g"] Oct 30 22:08:19.231564 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.231532 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3d5d13-68b98f4bcb-zhp9g"] Oct 30 22:08:19.349482 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.349453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:19.620384 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:19.620348 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:20.924740 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:20.924707 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cc7de8-a788-4d88-a3f0-d14270a16b7b" path="/var/lib/kubelet/pods/f3cc7de8-a788-4d88-a3f0-d14270a16b7b/volumes" Oct 30 22:08:21.095431 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:21.095381 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:23.147749 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:23.147709 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:23.593548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:23.593466 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:23.754979 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:23.754941 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:23.851301 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:23.851224 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps"] Oct 30 22:08:23.851831 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:23.851792 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" containerID="cri-o://9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8" gracePeriod=30 Oct 30 22:08:23.937261 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:23.937232 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:25.120621 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:25.120576 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:25.184932 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:25.184897 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:08:26.155124 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.155078 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:26.342053 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.342012 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:26.423163 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.423079 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh"] Oct 30 22:08:26.423727 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.423693 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" containerID="cri-o://21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064" gracePeriod=30 Oct 30 22:08:26.454910 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.454882 5039 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:08:26.455023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.454942 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:08:26.519862 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:26.519818 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:27.071278 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:27.071242 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:28.368634 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:28.368601 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:08:28.369007 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:28.368667 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:08:29.005333 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.004088 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:29.141014 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.140989 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 22:08:29.246728 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.246698 5039 generic.go:356] "Generic (PLEG): container finished" podID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerID="9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8" exitCode=0 Oct 30 22:08:29.246885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.246766 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" Oct 30 22:08:29.246885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.246782 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" event={"ID":"80b2f6b5-c4bf-46a0-b78e-58994839e3ad","Type":"ContainerDied","Data":"9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8"} Oct 30 22:08:29.246885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.246838 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps" event={"ID":"80b2f6b5-c4bf-46a0-b78e-58994839e3ad","Type":"ContainerDied","Data":"9f5b0540d7a7bfcaf1d2dd0a4f2235e26c028a5468fc4b899a1385a6cd1bafb8"} Oct 30 22:08:29.246885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.246862 5039 scope.go:122] "RemoveContainer" containerID="9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8" Oct 30 22:08:29.256121 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.256091 5039 scope.go:122] "RemoveContainer" containerID="3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7" Oct 30 22:08:29.266728 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.266701 5039 scope.go:122] "RemoveContainer" containerID="9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8" Oct 30 22:08:29.267555 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:29.266973 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8\": container with ID starting with 9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8 not found: ID does not exist" containerID="9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8" Oct 30 22:08:29.267555 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.267012 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8"} err="failed to get container status \"9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8\": rpc error: code = NotFound desc = could not find container \"9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8\": container with ID starting with 9832af10456ee0e24996f9edc4a8466749a8c4c8922a5fe3af58712273fba1f8 not found: ID does not exist" Oct 30 22:08:29.267555 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.267033 5039 scope.go:122] "RemoveContainer" containerID="3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7" Oct 30 22:08:29.267555 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:29.267321 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7\": container with ID starting with 3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7 not found: ID does not exist" containerID="3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7" Oct 30 22:08:29.267555 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.267352 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7"} err="failed to get container status \"3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7\": rpc error: code = NotFound desc = could not find container \"3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7\": container with ID starting with 3ea739ea69b3a657dd498fc8d009f81e06b485553d46667848c3877c8f111dc7 not found: ID does not exist" Oct 30 22:08:29.283213 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283194 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283307 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283262 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283353 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283306 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283417 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283368 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283479 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283425 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283547 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283507 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:29.283598 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283578 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283660 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283607 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:29.283660 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283646 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283771 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283739 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.283834 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.283817 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket\") pod \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\" (UID: \"80b2f6b5-c4bf-46a0-b78e-58994839e3ad\") " Oct 30 22:08:29.284089 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284065 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:29.284200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284137 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:29.284332 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284309 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:29.284565 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284546 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.284648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284571 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.284648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284584 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.284648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284597 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.284648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.284610 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.286017 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.285981 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:29.286114 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.286031 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:29.286169 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.286115 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "kube-api-access-txcfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:29.286243 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.286219 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo" pod "80b2f6b5-c4bf-46a0-b78e-58994839e3ad" (UID: "80b2f6b5-c4bf-46a0-b78e-58994839e3ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:29.385866 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.385834 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.385866 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.385864 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.386199 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.385879 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txcfr\" (UniqueName: \"kubernetes.io/projected/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-kube-api-access-txcfr\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.386199 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.385894 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2f6b5-c4bf-46a0-b78e-58994839e3ad-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:29.589703 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.589677 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps"] Oct 30 22:08:29.601973 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:29.601948 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-bq7ps"] Oct 30 22:08:30.783438 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:30.783378 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:30.922972 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:30.922936 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" path="/var/lib/kubelet/pods/80b2f6b5-c4bf-46a0-b78e-58994839e3ad/volumes" Oct 30 22:08:31.708373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.708318 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 22:08:31.814861 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.814834 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.814861 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.814864 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815329 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815015 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815329 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815069 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815329 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815162 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815329 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815219 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815329 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815284 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815336 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:31.815548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815453 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.815548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815464 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:31.815664 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815567 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:31.815734 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815621 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:31.815770 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815713 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:31.815770 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.815750 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy\") pod \"bb3404db-e958-4f7c-b4a2-8edad0eab199\" (UID: \"bb3404db-e958-4f7c-b4a2-8edad0eab199\") " Oct 30 22:08:31.816347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.816324 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.816484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.816352 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3404db-e958-4f7c-b4a2-8edad0eab199-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.816484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.816367 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.816484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.816378 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.816484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.816410 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.817202 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.817182 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "kube-api-access-sjf7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:31.817306 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.817210 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:31.817306 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.817220 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:31.817607 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.817593 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy" pod "bb3404db-e958-4f7c-b4a2-8edad0eab199" (UID: "bb3404db-e958-4f7c-b4a2-8edad0eab199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:31.916917 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.916893 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.916917 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.916913 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.917041 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.916922 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:31.917041 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:31.916931 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjf7v\" (UniqueName: \"kubernetes.io/projected/bb3404db-e958-4f7c-b4a2-8edad0eab199-kube-api-access-sjf7v\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:08:32.268483 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.268458 5039 generic.go:356] "Generic (PLEG): container finished" podID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerID="21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064" exitCode=0 Oct 30 22:08:32.268608 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.268503 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" event={"ID":"bb3404db-e958-4f7c-b4a2-8edad0eab199","Type":"ContainerDied","Data":"21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064"} Oct 30 22:08:32.268608 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.268555 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" event={"ID":"bb3404db-e958-4f7c-b4a2-8edad0eab199","Type":"ContainerDied","Data":"34d4f46d025f96b9fc0d9e6c04df40121c021f60707aaaef35faecb034fc9070"} Oct 30 22:08:32.268608 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.268576 5039 scope.go:122] "RemoveContainer" containerID="21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064" Oct 30 22:08:32.268608 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.268512 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh" Oct 30 22:08:32.280302 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.280283 5039 scope.go:122] "RemoveContainer" containerID="4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb" Oct 30 22:08:32.289796 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.289775 5039 scope.go:122] "RemoveContainer" containerID="21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064" Oct 30 22:08:32.290092 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:32.290071 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064\": container with ID starting with 21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064 not found: ID does not exist" containerID="21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064" Oct 30 22:08:32.290188 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.290096 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064"} err="failed to get container status \"21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064\": rpc error: code = NotFound desc = could not find container \"21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064\": container with ID starting with 21a6067a60ec1fb6134ccb9124d34682707eeba75407e7ee66716b2e19d0f064 not found: ID does not exist" Oct 30 22:08:32.290188 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.290112 5039 scope.go:122] "RemoveContainer" containerID="4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb" Oct 30 22:08:32.290376 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:32.290352 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb\": container with ID starting with 4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb not found: ID does not exist" containerID="4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb" Oct 30 22:08:32.290447 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.290387 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb"} err="failed to get container status \"4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb\": rpc error: code = NotFound desc = could not find container \"4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb\": container with ID starting with 4a9fa644e545bf8a96e4891c9b9cc6f799a416055da15ffed0952344504e3fdb not found: ID does not exist" Oct 30 22:08:32.326295 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.326274 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh"] Oct 30 22:08:32.335754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.335736 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3d5d13-openshift-default-76df5844f-bx8bh"] Oct 30 22:08:32.378931 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.378848 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:32.921809 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:32.921778 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" path="/var/lib/kubelet/pods/bb3404db-e958-4f7c-b4a2-8edad0eab199/volumes" Oct 30 22:08:34.000089 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:34.000045 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:35.685934 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:35.685897 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:37.315991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:37.315953 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:37.467916 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:37.467881 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:37.622567 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:37.622484 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:39.039575 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:39.039535 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:39.943478 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:39.943445 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:08:40.735768 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:40.735724 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:42.414528 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:42.414491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:42.741283 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:42.741212 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.037174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.037133 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.060530 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.060501 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn"] Oct 30 22:08:44.061100 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061082 5039 cpu_manager.go:414] "containerMap: removing container" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061102 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061117 5039 cpu_manager.go:414] "containerMap: removing container" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061124 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061161 5039 cpu_manager.go:414] "containerMap: removing container" podUID="3ed4bc48-36e8-4858-b38e-2f3dbf766df3" containerName="maas-api" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061168 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed4bc48-36e8-4858-b38e-2f3dbf766df3" containerName="maas-api" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061182 5039 cpu_manager.go:414] "containerMap: removing container" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" Oct 30 22:08:44.061200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061188 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061202 5039 cpu_manager.go:414] "containerMap: removing container" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061209 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061231 5039 cpu_manager.go:414] "containerMap: removing container" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061239 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061262 5039 cpu_manager.go:414] "containerMap: removing container" podUID="f3cc7de8-a788-4d88-a3f0-d14270a16b7b" containerName="maas-api" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061268 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cc7de8-a788-4d88-a3f0-d14270a16b7b" containerName="maas-api" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061284 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061291 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061316 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061323 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061476 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061489 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="66188eb2-2660-4cca-9fcf-e70428ee5286" containerName="main" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061501 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061515 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed4bc48-36e8-4858-b38e-2f3dbf766df3" containerName="maas-api" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061524 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b2f6b5-c4bf-46a0-b78e-58994839e3ad" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061535 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061545 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac72929-ed78-471f-8d8d-49f5dde68520" containerName="istio-proxy" Oct 30 22:08:44.061573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.061558 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cc7de8-a788-4d88-a3f0-d14270a16b7b" containerName="maas-api" Oct 30 22:08:44.065281 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.065261 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.090885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.090756 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5454af\"/\"openshift-service-ca.crt\"" Oct 30 22:08:44.091110 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.091090 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5454af\"/\"default-dockercfg-4gzwx\"" Oct 30 22:08:44.091231 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.091150 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5454af\"/\"test-model-5454af-kserve-self-signed-certs\"" Oct 30 22:08:44.091303 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.091237 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5454af\"/\"kube-root-ca.crt\"" Oct 30 22:08:44.094995 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.094970 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn"] Oct 30 22:08:44.152630 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.152597 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.152773 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.152756 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.152838 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.152817 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.152905 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.152864 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.153031 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.153012 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkgc\" (UniqueName: \"kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.153154 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.153127 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.207315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.207284 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.254269 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254243 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254405 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254318 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkgc\" (UniqueName: \"kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254405 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254353 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254534 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254409 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254534 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254517 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254652 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254554 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254708 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.254829 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.254808 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.255241 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.255214 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.256768 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.256745 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.257172 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.257151 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.270754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.270732 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkgc\" (UniqueName: \"kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc\") pod \"test-model-5454af-kserve-cb7c5848f-22mtn\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.379364 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.379022 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:44.722356 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:44.722254 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn"] Oct 30 22:08:45.380267 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:45.380228 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" event={"ID":"933a2d98-249d-4af7-a3f7-a73b8303410c","Type":"ContainerStarted","Data":"735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c"} Oct 30 22:08:45.380670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:45.380279 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" event={"ID":"933a2d98-249d-4af7-a3f7-a73b8303410c","Type":"ContainerStarted","Data":"81845d13a7af07a0972117fd6c5339824c4908131cf5edf5895959d91276430c"} Oct 30 22:08:45.417605 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:45.417564 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" podStartSLOduration=1.417550708 podStartE2EDuration="1.417550708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:45.417062722 +0000 UTC m=+3564.627002350" watchObservedRunningTime="2026-10-30 22:08:45.417550708 +0000 UTC m=+3564.627490369" Oct 30 22:08:45.572890 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:45.572852 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.380599 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:46.380572 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:47.141806 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:47.141761 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:48.845825 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:48.841739 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:50.532844 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:50.532798 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:52.288373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:52.288335 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:53.967325 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:53.967286 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:54.380333 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:54.380256 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:54.392064 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:54.392040 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:54.444503 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:54.444480 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:08:55.184468 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:08:55.184434 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:08:55.549588 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:55.549470 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:57.224712 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:57.224655 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.801942 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:08:58.801850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:00.590362 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:00.590324 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:02.361556 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:02.361517 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.083894 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.083286 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.458576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.458546 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v"] Oct 30 22:09:04.459207 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.459192 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3404db-e958-4f7c-b4a2-8edad0eab199" containerName="istio-proxy" Oct 30 22:09:04.462713 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.462690 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.493939 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.493870 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-589ebc\"/\"test-model-589ebc-kserve-self-signed-certs\"" Oct 30 22:09:04.493939 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.493882 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-589ebc\"/\"kube-root-ca.crt\"" Oct 30 22:09:04.493939 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.493885 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-589ebc\"/\"default-dockercfg-l49f2\"" Oct 30 22:09:04.493939 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.493933 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-589ebc\"/\"openshift-service-ca.crt\"" Oct 30 22:09:04.498453 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.498430 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v"] Oct 30 22:09:04.578992 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.578960 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.579159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.579039 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frp6\" (UniqueName: \"kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.579242 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.579151 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.579301 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.579241 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.579301 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.579278 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.579495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.579369 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.601194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.601155 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.681107 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681072 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681251 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681143 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681251 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681210 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681368 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681283 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681372 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frp6\" (UniqueName: \"kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681497 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681475 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681618 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681589 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681788 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681764 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.681854 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.681811 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.683447 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.683384 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.684002 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.683981 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.698950 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.698930 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frp6\" (UniqueName: \"kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6\") pod \"test-model-589ebc-kserve-849cc9fcc5-8mh7v\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:04.776150 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:04.775829 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:05.130430 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:05.130386 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v"] Oct 30 22:09:05.131199 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:09:05.131174 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94a805cf_6f02_4e24_9205_1f729fc615fa.slice/crio-a7ab14c3bd4aefa21a28ab47e7af3ad3e6532d0844977c4c30b633621feac3df WatchSource:0}: Error finding container a7ab14c3bd4aefa21a28ab47e7af3ad3e6532d0844977c4c30b633621feac3df: Status 404 returned error can't find the container with id a7ab14c3bd4aefa21a28ab47e7af3ad3e6532d0844977c4c30b633621feac3df Oct 30 22:09:05.507573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:05.507540 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" event={"ID":"94a805cf-6f02-4e24-9205-1f729fc615fa","Type":"ContainerStarted","Data":"ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023"} Oct 30 22:09:05.507732 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:05.507585 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" event={"ID":"94a805cf-6f02-4e24-9205-1f729fc615fa","Type":"ContainerStarted","Data":"a7ab14c3bd4aefa21a28ab47e7af3ad3e6532d0844977c4c30b633621feac3df"} Oct 30 22:09:05.548686 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:05.548612 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" podStartSLOduration=1.548598669 podStartE2EDuration="1.548598669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:05.547117059 +0000 UTC m=+3584.757056687" watchObservedRunningTime="2026-10-30 22:09:05.548598669 +0000 UTC m=+3584.758538337" Oct 30 22:09:05.778439 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:05.778343 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:05.823264 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:05.823228 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:07.528008 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:07.527964 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:09.092326 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:09.092293 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:10.678838 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:10.678796 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:12.401868 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:12.401824 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:13.645031 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:13.644999 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh"] Oct 30 22:09:13.645439 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:13.645181 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" podUID="fa1bb836-9911-4a78-9534-214c4916bf49" containerName="maas-api" containerID="cri-o://6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c" gracePeriod=30 Oct 30 22:09:13.776308 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:13.776273 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:14.015436 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.014779 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 22:09:14.149424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.146959 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls\") pod \"fa1bb836-9911-4a78-9534-214c4916bf49\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " Oct 30 22:09:14.149424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.147179 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca\") pod \"fa1bb836-9911-4a78-9534-214c4916bf49\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " Oct 30 22:09:14.149424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.147228 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn\" (UniqueName: \"kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn\") pod \"fa1bb836-9911-4a78-9534-214c4916bf49\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " Oct 30 22:09:14.149424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.147364 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs\") pod \"fa1bb836-9911-4a78-9534-214c4916bf49\" (UID: \"fa1bb836-9911-4a78-9534-214c4916bf49\") " Oct 30 22:09:14.149424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.147676 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca" pod "fa1bb836-9911-4a78-9534-214c4916bf49" (UID: "fa1bb836-9911-4a78-9534-214c4916bf49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:14.149424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.148197 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1bb836-9911-4a78-9534-214c4916bf49-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:14.154640 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.150909 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn" pod "fa1bb836-9911-4a78-9534-214c4916bf49" (UID: "fa1bb836-9911-4a78-9534-214c4916bf49"). InnerVolumeSpecName "kube-api-access-wbghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:14.154640 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.151042 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls" pod "fa1bb836-9911-4a78-9534-214c4916bf49" (UID: "fa1bb836-9911-4a78-9534-214c4916bf49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:14.154640 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.153045 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs" pod "fa1bb836-9911-4a78-9534-214c4916bf49" (UID: "fa1bb836-9911-4a78-9534-214c4916bf49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:14.249299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.249228 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:14.249299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.249256 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbghn\" (UniqueName: \"kubernetes.io/projected/fa1bb836-9911-4a78-9534-214c4916bf49-kube-api-access-wbghn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:14.249299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.249265 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1bb836-9911-4a78-9534-214c4916bf49-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:14.458672 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.458641 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:14.584958 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.584882 5039 generic.go:356] "Generic (PLEG): container finished" podID="fa1bb836-9911-4a78-9534-214c4916bf49" containerID="6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c" exitCode=0 Oct 30 22:09:14.584958 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.584926 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" Oct 30 22:09:14.585169 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.584926 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" event={"ID":"fa1bb836-9911-4a78-9534-214c4916bf49","Type":"ContainerDied","Data":"6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c"} Oct 30 22:09:14.585169 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.585041 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh" event={"ID":"fa1bb836-9911-4a78-9534-214c4916bf49","Type":"ContainerDied","Data":"1a554b198ab93f429c855df6346634448c01be78ca892cea7b1f7c4ad8dc759d"} Oct 30 22:09:14.585169 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.585071 5039 scope.go:122] "RemoveContainer" containerID="6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c" Oct 30 22:09:14.595474 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.595453 5039 scope.go:122] "RemoveContainer" containerID="6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c" Oct 30 22:09:14.595714 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:14.595698 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c\": container with ID starting with 6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c not found: ID does not exist" containerID="6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c" Oct 30 22:09:14.595762 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.595722 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c"} err="failed to get container status \"6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c\": rpc error: code = NotFound desc = could not find container \"6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c\": container with ID starting with 6c627ddc72ceb60b6d4ae0c0262dcb81eef1e684601965d1e3a21fbb781e0e3c not found: ID does not exist" Oct 30 22:09:14.622550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.622513 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:14.634550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.634517 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh"] Oct 30 22:09:14.645527 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.645497 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdb168-659c45c584-j6tsh"] Oct 30 22:09:14.777386 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.777356 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:14.778092 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.778076 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:14.789667 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.789647 5039 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:14.922741 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:14.922720 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1bb836-9911-4a78-9534-214c4916bf49" path="/var/lib/kubelet/pods/fa1bb836-9911-4a78-9534-214c4916bf49/volumes" Oct 30 22:09:15.051725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:15.051694 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.602806 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:15.602782 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:09:16.315281 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:16.314802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:17.214930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:17.214901 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq"] Oct 30 22:09:17.215120 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:17.215082 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" containerID="cri-o://7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97" gracePeriod=30 Oct 30 22:09:17.579415 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:17.579302 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:17.885453 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:17.885419 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.508412 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:19.508137 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:21.263635 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.263605 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:21.470415 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.456056 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:21.525115 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.525036 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:09:21.529113 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.529087 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:09:21.617222 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.617189 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb"] Oct 30 22:09:21.617493 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.617460 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" containerID="cri-o://49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8" gracePeriod=30 Oct 30 22:09:21.692597 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:21.692561 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:22.115122 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.115089 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:22.489734 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.489709 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:09:22.560766 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.560735 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.560923 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.560803 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.560923 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.560859 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561044 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.560985 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561044 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561012 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.561147 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561068 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561147 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561142 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561267 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561246 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561336 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561319 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561469 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561447 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert\") pod \"eba35083-ab20-4a9d-a265-1238b03c7d07\" (UID: \"eba35083-ab20-4a9d-a265-1238b03c7d07\") " Oct 30 22:09:22.561833 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561456 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.561833 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561732 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.561833 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561744 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.561988 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.561932 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:22.562369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.562267 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba35083-ab20-4a9d-a265-1238b03c7d07-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.562369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.562291 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.562369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.562304 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.562369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.562319 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.562369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.562332 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.564174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.564153 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:22.564338 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.564314 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "kube-api-access-qjf5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:22.564595 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.564571 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:22.567490 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.567471 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo" pod "eba35083-ab20-4a9d-a265-1238b03c7d07" (UID: "eba35083-ab20-4a9d-a265-1238b03c7d07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:22.641137 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.641081 5039 generic.go:356] "Generic (PLEG): container finished" podID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerID="7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97" exitCode=0 Oct 30 22:09:22.641239 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.641134 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" Oct 30 22:09:22.641239 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.641160 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" event={"ID":"eba35083-ab20-4a9d-a265-1238b03c7d07","Type":"ContainerDied","Data":"7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97"} Oct 30 22:09:22.641239 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.641207 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq" event={"ID":"eba35083-ab20-4a9d-a265-1238b03c7d07","Type":"ContainerDied","Data":"8338ef937cc3cf3c4150870e5d61f2b5a6d60105a7ca7b377f124485a1042322"} Oct 30 22:09:22.641239 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.641229 5039 scope.go:122] "RemoveContainer" containerID="7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97" Oct 30 22:09:22.655738 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.655716 5039 scope.go:122] "RemoveContainer" containerID="7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97" Oct 30 22:09:22.656027 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:22.656007 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97\": container with ID starting with 7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97 not found: ID does not exist" containerID="7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97" Oct 30 22:09:22.656091 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.656040 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97"} err="failed to get container status \"7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97\": rpc error: code = NotFound desc = could not find container \"7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97\": container with ID starting with 7375e6c1244a42e1b5c905cb1d65eb73ea743908dae0f9619a86d87db6d1ac97 not found: ID does not exist" Oct 30 22:09:22.663938 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.663914 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.664027 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.663939 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjf5l\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-kube-api-access-qjf5l\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.664027 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.663953 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.664027 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.663964 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba35083-ab20-4a9d-a265-1238b03c7d07-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:22.697312 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.697285 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq"] Oct 30 22:09:22.699434 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.699410 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-88zsq"] Oct 30 22:09:22.923183 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.923117 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" path="/var/lib/kubelet/pods/eba35083-ab20-4a9d-a265-1238b03c7d07/volumes" Oct 30 22:09:22.961070 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.961044 5039 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:22.961186 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:22.961102 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:23.039416 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:23.035104 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:24.800852 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:24.800513 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:25.195810 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:25.195766 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:09:26.345037 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.345005 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:26.878740 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.878717 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 22:09:26.914308 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914288 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914433 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914330 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914433 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914380 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914546 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914450 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914602 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914558 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914602 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914588 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:26.914693 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914681 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914749 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914721 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914679 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:26.914798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914762 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:26.914798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914786 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.914944 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914813 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:26.914944 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.914847 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket\") pod \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\" (UID: \"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f\") " Oct 30 22:09:26.915180 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.915153 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:26.915312 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.915298 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:26.915376 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.915317 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:26.915376 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.915331 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:26.915376 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.915344 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:26.915376 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.915357 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:26.917258 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.917232 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:26.917373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.917263 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:26.917373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.917268 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "kube-api-access-hjhnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:26.917373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:26.917280 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy" pod "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" (UID: "2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:27.016256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.016231 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:27.016256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.016254 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:27.016256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.016262 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:27.016501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.016271 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjhnj\" (UniqueName: \"kubernetes.io/projected/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f-kube-api-access-hjhnj\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:27.052334 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.052309 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h"] Oct 30 22:09:27.060545 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.060522 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296"] Oct 30 22:09:27.204814 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.204727 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:27.686714 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.686669 5039 generic.go:356] "Generic (PLEG): container finished" podID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerID="49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8" exitCode=0 Oct 30 22:09:27.687159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.686742 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" Oct 30 22:09:27.687159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.686738 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" event={"ID":"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f","Type":"ContainerDied","Data":"49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8"} Oct 30 22:09:27.687159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.686905 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb" event={"ID":"2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f","Type":"ContainerDied","Data":"65369e8e07c210e73efdd28176eb2e0bb50dacbaf23a99dafed7e8f6454dbefa"} Oct 30 22:09:27.687159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.686941 5039 scope.go:122] "RemoveContainer" containerID="49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8" Oct 30 22:09:27.696715 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.696697 5039 scope.go:122] "RemoveContainer" containerID="5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f" Oct 30 22:09:27.705205 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.705189 5039 scope.go:122] "RemoveContainer" containerID="49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8" Oct 30 22:09:27.705465 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:27.705443 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8\": container with ID starting with 49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8 not found: ID does not exist" containerID="49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8" Oct 30 22:09:27.705563 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.705469 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8"} err="failed to get container status \"49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8\": rpc error: code = NotFound desc = could not find container \"49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8\": container with ID starting with 49854ebb4a2f0a5d3f3dc08f2ada6262fa58fc8dec561fe07f2dda895beffdb8 not found: ID does not exist" Oct 30 22:09:27.705563 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.705484 5039 scope.go:122] "RemoveContainer" containerID="5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f" Oct 30 22:09:27.705714 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:27.705701 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f\": container with ID starting with 5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f not found: ID does not exist" containerID="5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f" Oct 30 22:09:27.705757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.705716 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f"} err="failed to get container status \"5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f\": rpc error: code = NotFound desc = could not find container \"5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f\": container with ID starting with 5f45cd3955c55afccd46bb2e14a3d6557e096edccdee6d298aea0bb8dc30f97f not found: ID does not exist" Oct 30 22:09:27.741622 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.741599 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb"] Oct 30 22:09:27.751429 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:27.751405 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-s5kfb"] Oct 30 22:09:28.028328 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:28.028252 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:28.922062 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:28.922021 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" path="/var/lib/kubelet/pods/2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f/volumes" Oct 30 22:09:29.573575 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:29.573532 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:29.821931 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:29.821898 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:09:31.436042 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:31.435962 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:32.955400 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:32.955360 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:33.100049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:33.100018 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:33.250176 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:33.250092 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:34.885912 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:34.885869 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:36.507118 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:36.507075 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:38.081986 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:38.081951 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:39.609299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:39.609265 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.270412 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.270353 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg"] Oct 30 22:09:40.270588 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.270549 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" podUID="182ef68a-10a3-4119-a059-3883a088fe64" containerName="discovery" containerID="cri-o://868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3" gracePeriod=30 Oct 30 22:09:40.547811 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.547787 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:09:40.663644 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.663608 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.663710 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.663767 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.663825 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.663881 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.663986 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.664016 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert\") pod \"182ef68a-10a3-4119-a059-3883a088fe64\" (UID: \"182ef68a-10a3-4119-a059-3883a088fe64\") " Oct 30 22:09:40.664365 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.664190 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:40.664611 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.664587 5039 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-ca-configmap\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.666354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.666304 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:40.666354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.666323 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:40.666574 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.666431 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "kube-api-access-bwvqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:40.666574 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.666456 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:40.666574 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.666513 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:40.666574 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.666555 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs" pod "182ef68a-10a3-4119-a059-3883a088fe64" (UID: "182ef68a-10a3-4119-a059-3883a088fe64"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:40.765699 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.765670 5039 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-cacerts\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.765699 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.765696 5039 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-csr-dns-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.765855 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.765706 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.765855 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.765716 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvqp\" (UniqueName: \"kubernetes.io/projected/182ef68a-10a3-4119-a059-3883a088fe64-kube-api-access-bwvqp\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.765855 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.765725 5039 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ef68a-10a3-4119-a059-3883a088fe64-istio-kubeconfig\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.765855 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.765733 5039 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ef68a-10a3-4119-a059-3883a088fe64-local-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:40.771763 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.771736 5039 generic.go:356] "Generic (PLEG): container finished" podID="182ef68a-10a3-4119-a059-3883a088fe64" containerID="868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3" exitCode=0 Oct 30 22:09:40.771875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.771786 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" Oct 30 22:09:40.771875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.771828 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" event={"ID":"182ef68a-10a3-4119-a059-3883a088fe64","Type":"ContainerDied","Data":"868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3"} Oct 30 22:09:40.771963 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.771882 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg" event={"ID":"182ef68a-10a3-4119-a059-3883a088fe64","Type":"ContainerDied","Data":"69e4e731ff2f35a14931848c8603d5bd563493242359abc2ff1e4ef0725d758f"} Oct 30 22:09:40.771963 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.771906 5039 scope.go:122] "RemoveContainer" containerID="868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3" Oct 30 22:09:40.783174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.783160 5039 scope.go:122] "RemoveContainer" containerID="868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3" Oct 30 22:09:40.783421 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:40.783386 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3\": container with ID starting with 868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3 not found: ID does not exist" containerID="868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3" Oct 30 22:09:40.783493 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.783433 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3"} err="failed to get container status \"868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3\": rpc error: code = NotFound desc = could not find container \"868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3\": container with ID starting with 868a411cdc78b2e74ba6f3836dc318ed33f7cf379cea0f3d1867146b1cc446c3 not found: ID does not exist" Oct 30 22:09:40.817096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.817077 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg"] Oct 30 22:09:40.820691 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.820670 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-srlwg"] Oct 30 22:09:40.921542 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:40.921513 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182ef68a-10a3-4119-a059-3883a088fe64" path="/var/lib/kubelet/pods/182ef68a-10a3-4119-a059-3883a088fe64/volumes" Oct 30 22:09:41.288544 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:41.288503 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:42.072801 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.072769 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" containerID="cri-o://4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853" gracePeriod=45 Oct 30 22:09:42.081253 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.081203 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" containerID="cri-o://ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb" gracePeriod=45 Oct 30 22:09:42.349317 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.349294 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 22:09:42.373496 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.373476 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 22:09:42.485317 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485283 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir\") pod \"436b8ce8-13a5-4b79-bd40-868bfb93947b\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " Oct 30 22:09:42.485531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485345 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir\") pod \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " Oct 30 22:09:42.485531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485447 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm\") pod \"436b8ce8-13a5-4b79-bd40-868bfb93947b\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " Oct 30 22:09:42.485531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485499 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l\" (UniqueName: \"kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l\") pod \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " Oct 30 22:09:42.485879 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485579 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm\") pod \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " Oct 30 22:09:42.485879 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485624 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs\" (UniqueName: \"kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs\") pod \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " Oct 30 22:09:42.485879 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485652 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache\") pod \"436b8ce8-13a5-4b79-bd40-868bfb93947b\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " Oct 30 22:09:42.486081 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.485704 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home\") pod \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " Oct 30 22:09:42.486183 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.486165 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache\") pod \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\" (UID: \"93938b5f-db5a-4a74-b3a3-7c7fd42c8362\") " Oct 30 22:09:42.486251 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.486237 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs\" (UniqueName: \"kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs\") pod \"436b8ce8-13a5-4b79-bd40-868bfb93947b\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " Oct 30 22:09:42.486319 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.486303 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home\") pod \"436b8ce8-13a5-4b79-bd40-868bfb93947b\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " Oct 30 22:09:42.486376 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.486356 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir" pod "93938b5f-db5a-4a74-b3a3-7c7fd42c8362" (UID: "93938b5f-db5a-4a74-b3a3-7c7fd42c8362"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.487562 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.486374 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5\" (UniqueName: \"kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5\") pod \"436b8ce8-13a5-4b79-bd40-868bfb93947b\" (UID: \"436b8ce8-13a5-4b79-bd40-868bfb93947b\") " Oct 30 22:09:42.488029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.487753 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.488231 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.488204 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache" pod "93938b5f-db5a-4a74-b3a3-7c7fd42c8362" (UID: "93938b5f-db5a-4a74-b3a3-7c7fd42c8362"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.488473 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.488452 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache" pod "436b8ce8-13a5-4b79-bd40-868bfb93947b" (UID: "436b8ce8-13a5-4b79-bd40-868bfb93947b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.488675 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.488650 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home" pod "93938b5f-db5a-4a74-b3a3-7c7fd42c8362" (UID: "93938b5f-db5a-4a74-b3a3-7c7fd42c8362"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.488776 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.488686 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm" pod "93938b5f-db5a-4a74-b3a3-7c7fd42c8362" (UID: "93938b5f-db5a-4a74-b3a3-7c7fd42c8362"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.488952 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.488925 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home" pod "436b8ce8-13a5-4b79-bd40-868bfb93947b" (UID: "436b8ce8-13a5-4b79-bd40-868bfb93947b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.489085 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.489067 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir" pod "436b8ce8-13a5-4b79-bd40-868bfb93947b" (UID: "436b8ce8-13a5-4b79-bd40-868bfb93947b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.490760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.490736 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs" pod "436b8ce8-13a5-4b79-bd40-868bfb93947b" (UID: "436b8ce8-13a5-4b79-bd40-868bfb93947b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:42.493319 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.493295 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5" pod "436b8ce8-13a5-4b79-bd40-868bfb93947b" (UID: "436b8ce8-13a5-4b79-bd40-868bfb93947b"). InnerVolumeSpecName "kube-api-access-m65q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:42.493499 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.493475 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs" pod "93938b5f-db5a-4a74-b3a3-7c7fd42c8362" (UID: "93938b5f-db5a-4a74-b3a3-7c7fd42c8362"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:42.493499 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.493489 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm" pod "436b8ce8-13a5-4b79-bd40-868bfb93947b" (UID: "436b8ce8-13a5-4b79-bd40-868bfb93947b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:42.495597 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.495572 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l" pod "93938b5f-db5a-4a74-b3a3-7c7fd42c8362" (UID: "93938b5f-db5a-4a74-b3a3-7c7fd42c8362"). InnerVolumeSpecName "kube-api-access-xgp8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588534 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588562 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588570 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgp8l\" (UniqueName: \"kubernetes.io/projected/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-kube-api-access-xgp8l\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588580 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588588 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588596 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588604 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588603 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588612 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93938b5f-db5a-4a74-b3a3-7c7fd42c8362-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588621 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436b8ce8-13a5-4b79-bd40-868bfb93947b-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588628 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/436b8ce8-13a5-4b79-bd40-868bfb93947b-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.588909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.588635 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65q5\" (UniqueName: \"kubernetes.io/projected/436b8ce8-13a5-4b79-bd40-868bfb93947b-kube-api-access-m65q5\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:09:42.785945 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.785911 5039 generic.go:356] "Generic (PLEG): container finished" podID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerID="4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853" exitCode=0 Oct 30 22:09:42.785945 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.785945 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" Oct 30 22:09:42.786179 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.785980 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" event={"ID":"436b8ce8-13a5-4b79-bd40-868bfb93947b","Type":"ContainerDied","Data":"4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853"} Oct 30 22:09:42.786179 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.786022 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h" event={"ID":"436b8ce8-13a5-4b79-bd40-868bfb93947b","Type":"ContainerDied","Data":"e1448dd4b343ff55fcb4f0125f91f144635c9a1fe0ab2724f4d22fd04a3df927"} Oct 30 22:09:42.786179 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.786041 5039 scope.go:122] "RemoveContainer" containerID="4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853" Oct 30 22:09:42.787582 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.787549 5039 generic.go:356] "Generic (PLEG): container finished" podID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerID="ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb" exitCode=0 Oct 30 22:09:42.787699 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.787589 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" Oct 30 22:09:42.787699 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.787628 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" event={"ID":"93938b5f-db5a-4a74-b3a3-7c7fd42c8362","Type":"ContainerDied","Data":"ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb"} Oct 30 22:09:42.787699 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.787665 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" event={"ID":"93938b5f-db5a-4a74-b3a3-7c7fd42c8362","Type":"ContainerDied","Data":"06bf908f4149fecc36f19dabb6473a611f85fcfceecaefac594d297e2a5a84de"} Oct 30 22:09:42.796130 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.796109 5039 scope.go:122] "RemoveContainer" containerID="4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853" Oct 30 22:09:42.796427 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:42.796378 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853\": container with ID starting with 4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853 not found: ID does not exist" containerID="4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853" Oct 30 22:09:42.796534 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.796438 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853"} err="failed to get container status \"4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853\": rpc error: code = NotFound desc = could not find container \"4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853\": container with ID starting with 4ee7783e4ad591435190fd47b483542c2a8137b22b7ba6b7d1360f2fc1f35853 not found: ID does not exist" Oct 30 22:09:42.796534 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.796460 5039 scope.go:122] "RemoveContainer" containerID="ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb" Oct 30 22:09:42.805115 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.805100 5039 scope.go:122] "RemoveContainer" containerID="ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb" Oct 30 22:09:42.805377 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:42.805359 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb\": container with ID starting with ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb not found: ID does not exist" containerID="ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb" Oct 30 22:09:42.805461 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.805383 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb"} err="failed to get container status \"ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb\": rpc error: code = NotFound desc = could not find container \"ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb\": container with ID starting with ce8bd1cd185d9840112d9e4164c7cd292b7245001d52945cac57dba12559aecb not found: ID does not exist" Oct 30 22:09:42.833500 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.833479 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h"] Oct 30 22:09:42.839081 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.839036 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/premium-w3-50ab77-kserve-85d7954875-mhx4h"] Oct 30 22:09:42.867157 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.867129 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296"] Oct 30 22:09:42.873133 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.873111 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296"] Oct 30 22:09:42.900289 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.900264 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:42.926546 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.925466 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" path="/var/lib/kubelet/pods/436b8ce8-13a5-4b79-bd40-868bfb93947b/volumes" Oct 30 22:09:42.926546 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:42.926238 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" path="/var/lib/kubelet/pods/93938b5f-db5a-4a74-b3a3-7c7fd42c8362/volumes" Oct 30 22:09:43.319017 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:43.318982 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/simulator-w3-50ab77-kserve-bb75d44d6-br296" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:44.423722 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:44.423686 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:46.119239 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:46.119201 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:47.705184 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:47.705138 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:49.368918 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:49.368884 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:50.929986 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:50.929896 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:52.523158 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:52.523112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:54.110342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:54.110300 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:55.184437 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:09:55.184382 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:09:55.697354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:55.694507 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:57.331145 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:57.331099 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:58.987375 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:09:58.987338 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:00.617237 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:00.617188 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:02.415073 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:02.415022 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:03.828170 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:03.828136 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.116629 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:04.116327 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:05.815516 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:05.815477 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:07.360086 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:07.359861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:09.124489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:09.124449 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:10.788609 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:10.788570 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.300975 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:11.300943 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.470932 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:11.470841 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:12.610503 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:12.610463 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:13.622592 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:13.622557 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:14.345023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:14.344993 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:16.152360 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:16.152313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:16.990635 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:16.990602 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:10:17.795501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:17.795466 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.900321 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:19.900279 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.425218 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:21.425188 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.712102 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:21.712026 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.969615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:21.969527 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:23.547128 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:23.547088 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:25.185092 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:10:25.185057 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:10:25.242872 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:25.242831 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:26.789164 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:26.789123 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.326065 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:28.326012 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:29.925514 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:29.925477 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:31.763383 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:31.763339 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:33.239783 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:33.239748 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:33.527807 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:33.527711 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:34.856982 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:34.856941 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:36.363754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:36.363716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:37.982681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:37.982612 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:39.510117 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:39.510059 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:41.310572 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:41.310540 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:42.942802 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:42.942766 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:44.640603 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:44.640575 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:45.300361 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:45.300320 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:45.470025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:45.469997 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:45.611766 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:45.611694 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:46.042676 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:46.042640 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:47.320304 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:47.320260 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:49.244856 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:49.244822 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:50.873718 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:50.873672 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.597436 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:51.594677 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.667583 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:52.667543 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.828331 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:52.828292 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.950052 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:52.949976 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq"] Oct 30 22:10:52.950194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:52.950172 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" containerID="cri-o://6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8" gracePeriod=30 Oct 30 22:10:53.011874 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:53.011844 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:54.883058 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:54.883027 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:55.184975 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:10:55.184938 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:10:56.574614 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:56.574582 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.215506 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.215482 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 22:10:58.292996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.292914 5039 generic.go:356] "Generic (PLEG): container finished" podID="eb133100-05d0-4614-b31a-6f9851985807" containerID="6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8" exitCode=0 Oct 30 22:10:58.292996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.292964 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" Oct 30 22:10:58.293197 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.293139 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" event={"ID":"eb133100-05d0-4614-b31a-6f9851985807","Type":"ContainerDied","Data":"6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8"} Oct 30 22:10:58.293244 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.293222 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq" event={"ID":"eb133100-05d0-4614-b31a-6f9851985807","Type":"ContainerDied","Data":"1bfd109203f79f08936e89e059b9cee8252d0aa6648909540574c79f77af9f7b"} Oct 30 22:10:58.293289 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.293255 5039 scope.go:122] "RemoveContainer" containerID="6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8" Oct 30 22:10:58.296015 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.295989 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296041 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296079 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296116 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296143 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296198 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296279 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:58.296497 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296348 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296497 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296433 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296611 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296514 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert\") pod \"eb133100-05d0-4614-b31a-6f9851985807\" (UID: \"eb133100-05d0-4614-b31a-6f9851985807\") " Oct 30 22:10:58.296611 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.296548 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:58.297152 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.297128 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.297254 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.297156 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.297434 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.297382 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:58.297870 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.297848 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:58.298132 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.298111 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:58.298991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.298953 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:58.299432 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.299375 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "kube-api-access-pdjq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:58.300187 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.300164 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:58.300431 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.300377 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo" pod "eb133100-05d0-4614-b31a-6f9851985807" (UID: "eb133100-05d0-4614-b31a-6f9851985807"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:58.304738 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.304719 5039 scope.go:122] "RemoveContainer" containerID="6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8" Oct 30 22:10:58.305053 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:10:58.305011 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8\": container with ID starting with 6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8 not found: ID does not exist" containerID="6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8" Oct 30 22:10:58.305138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.305055 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8"} err="failed to get container status \"6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8\": rpc error: code = NotFound desc = could not find container \"6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8\": container with ID starting with 6efb06ccaf5afffc034021ddfa60171b1377b2603ba25587626f2bb567464ae8 not found: ID does not exist" Oct 30 22:10:58.398544 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398505 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjq4\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-kube-api-access-pdjq4\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.398544 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398540 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb133100-05d0-4614-b31a-6f9851985807-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.398813 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398557 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb133100-05d0-4614-b31a-6f9851985807-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.398813 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398568 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.398813 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398581 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.398813 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398592 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb133100-05d0-4614-b31a-6f9851985807-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.398813 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.398604 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb133100-05d0-4614-b31a-6f9851985807-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:10:58.437678 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.437635 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.649150 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.649112 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq"] Oct 30 22:10:58.658652 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.658624 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-9hbnq"] Oct 30 22:10:58.922297 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:10:58.922269 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb133100-05d0-4614-b31a-6f9851985807" path="/var/lib/kubelet/pods/eb133100-05d0-4614-b31a-6f9851985807/volumes" Oct 30 22:11:00.420294 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:00.420256 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.167874 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:02.167825 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.547281 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:02.547197 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.709669 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:02.709631 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.512070 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:03.512034 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf"] Oct 30 22:11:03.512505 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:03.512245 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" containerID="cri-o://1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182" gracePeriod=30 Oct 30 22:11:04.113046 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:04.113009 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.798576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:05.798536 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:07.724051 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:07.724013 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:08.769248 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.769213 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 22:11:08.910589 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910560 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910652 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910696 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910718 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910851 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910757 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910851 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910783 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910851 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910834 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.910984 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910890 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910984 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910926 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.910984 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910954 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket\") pod \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\" (UID: \"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76\") " Oct 30 22:11:08.911238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.910951 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.911238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911104 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.911238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911134 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:08.911479 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911245 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.911615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911533 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:08.911615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911559 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:08.911615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911571 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:08.911615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911583 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:08.911615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.911594 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:08.913161 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.913134 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:08.913244 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.913183 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "kube-api-access-l7z85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:08.913244 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.913202 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.913319 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:08.913298 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token" pod "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" (UID: "22cc53e2-1f21-48a7-b95a-4d0aab2f3d76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:09.013123 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.013094 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:09.013123 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.013122 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7z85\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-kube-api-access-l7z85\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:09.013123 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.013131 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:09.013426 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.013140 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:11:09.361612 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.361521 5039 generic.go:356] "Generic (PLEG): container finished" podID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerID="1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182" exitCode=0 Oct 30 22:11:09.361612 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.361568 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" Oct 30 22:11:09.361612 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.361599 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" event={"ID":"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76","Type":"ContainerDied","Data":"1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182"} Oct 30 22:11:09.361867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.361655 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf" event={"ID":"22cc53e2-1f21-48a7-b95a-4d0aab2f3d76","Type":"ContainerDied","Data":"ea1408df77be26910b0bf4cb75dc58913448f9e85f7daae7f701b1f1874674ae"} Oct 30 22:11:09.361867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.361685 5039 scope.go:122] "RemoveContainer" containerID="1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182" Oct 30 22:11:09.371485 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.371466 5039 scope.go:122] "RemoveContainer" containerID="1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182" Oct 30 22:11:09.371740 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:11:09.371720 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182\": container with ID starting with 1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182 not found: ID does not exist" containerID="1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182" Oct 30 22:11:09.371786 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.371747 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182"} err="failed to get container status \"1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182\": rpc error: code = NotFound desc = could not find container \"1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182\": container with ID starting with 1c4b25e4a7983757053c6d8c06aef410882f48f57a5b565905d9375f5b383182 not found: ID does not exist" Oct 30 22:11:09.419892 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.419869 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf"] Oct 30 22:11:09.429314 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.429294 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-z8jbf"] Oct 30 22:11:09.807288 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:09.807253 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.046870 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:10.046837 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.921164 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:10.921122 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" path="/var/lib/kubelet/pods/22cc53e2-1f21-48a7-b95a-4d0aab2f3d76/volumes" Oct 30 22:11:11.660582 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:11.660551 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:12.492970 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:12.492940 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:11:13.466414 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:13.465624 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:15.162947 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:15.162912 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.963250 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:16.963218 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:17.965421 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:17.965365 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:18.748310 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:18.748274 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:20.545344 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:20.545308 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.172655 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:22.172604 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:23.933731 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:23.933675 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:25.186447 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:11:25.186377 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:11:25.634479 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:25.634372 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:27.557827 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:27.557789 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:29.631200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:29.631156 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:31.290999 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:31.290960 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:32.689618 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:32.689517 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:32.863158 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:32.863128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:33.010016 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:33.009934 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:34.436104 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:34.436064 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:34.932776 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:34.932721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:35.073915 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:35.073882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:35.546022 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:35.545988 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:36.921047 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:36.921010 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:38.775932 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:38.775880 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:40.411192 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:40.411102 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:41.951463 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:41.951428 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:42.100146 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:42.100095 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:42.324565 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:42.324453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.857155 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:43.857114 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:45.482294 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:45.482261 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:47.085380 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:47.085336 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:47.761076 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:47.761036 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:11:48.707822 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:48.707778 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:50.304157 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:50.304112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:51.988945 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:51.988903 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.516883 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:53.516851 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.675913 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:53.675882 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.831237 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:53.831150 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:55.185597 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:11:55.185557 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:11:55.316975 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:55.316940 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:57.301166 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:57.301117 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.998858 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:11:58.998817 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.278917 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:00.278879 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn"] Oct 30 22:12:00.408448 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:00.408410 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.668367 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:00.668327 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:02.439498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:02.439414 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:04.111068 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:04.111029 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:05.902123 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:05.902079 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:07.401760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:07.401725 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:09.008780 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:09.008677 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:10.689533 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:10.689456 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:12.274988 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:12.274932 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:13.138740 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:13.138708 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:12:14.055931 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:14.055875 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:15.298357 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.298321 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" podUID="933a2d98-249d-4af7-a3f7-a73b8303410c" containerName="main" containerID="cri-o://735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c" gracePeriod=45 Oct 30 22:12:15.435237 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.435205 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" podUID="933a2d98-249d-4af7-a3f7-a73b8303410c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.155:8000/ready\": dial tcp 10.132.0.155:8000: connect: connection refused" Oct 30 22:12:15.591905 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.591877 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:12:15.665826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.665739 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:15.707713 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.707683 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir\") pod \"933a2d98-249d-4af7-a3f7-a73b8303410c\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " Oct 30 22:12:15.707856 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.707782 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc\" (UniqueName: \"kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc\") pod \"933a2d98-249d-4af7-a3f7-a73b8303410c\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " Oct 30 22:12:15.707979 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.707953 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir" pod "933a2d98-249d-4af7-a3f7-a73b8303410c" (UID: "933a2d98-249d-4af7-a3f7-a73b8303410c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:15.708027 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.707971 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home\") pod \"933a2d98-249d-4af7-a3f7-a73b8303410c\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " Oct 30 22:12:15.708027 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708021 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs\" (UniqueName: \"kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs\") pod \"933a2d98-249d-4af7-a3f7-a73b8303410c\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " Oct 30 22:12:15.708201 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708184 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache\") pod \"933a2d98-249d-4af7-a3f7-a73b8303410c\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " Oct 30 22:12:15.708271 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708228 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm\") pod \"933a2d98-249d-4af7-a3f7-a73b8303410c\" (UID: \"933a2d98-249d-4af7-a3f7-a73b8303410c\") " Oct 30 22:12:15.708271 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708229 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home" pod "933a2d98-249d-4af7-a3f7-a73b8303410c" (UID: "933a2d98-249d-4af7-a3f7-a73b8303410c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:15.708725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708675 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:15.708725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708703 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:15.708909 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.708767 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache" pod "933a2d98-249d-4af7-a3f7-a73b8303410c" (UID: "933a2d98-249d-4af7-a3f7-a73b8303410c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:15.710118 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.710091 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc" pod "933a2d98-249d-4af7-a3f7-a73b8303410c" (UID: "933a2d98-249d-4af7-a3f7-a73b8303410c"). InnerVolumeSpecName "kube-api-access-clkgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:15.710550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.710527 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm" pod "933a2d98-249d-4af7-a3f7-a73b8303410c" (UID: "933a2d98-249d-4af7-a3f7-a73b8303410c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:15.710550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.710539 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs" pod "933a2d98-249d-4af7-a3f7-a73b8303410c" (UID: "933a2d98-249d-4af7-a3f7-a73b8303410c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:15.754142 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.754099 5039 generic.go:356] "Generic (PLEG): container finished" podID="933a2d98-249d-4af7-a3f7-a73b8303410c" containerID="735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c" exitCode=0 Oct 30 22:12:15.754256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.754140 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" Oct 30 22:12:15.754256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.754164 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" event={"ID":"933a2d98-249d-4af7-a3f7-a73b8303410c","Type":"ContainerDied","Data":"735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c"} Oct 30 22:12:15.754256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.754208 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn" event={"ID":"933a2d98-249d-4af7-a3f7-a73b8303410c","Type":"ContainerDied","Data":"81845d13a7af07a0972117fd6c5339824c4908131cf5edf5895959d91276430c"} Oct 30 22:12:15.754256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.754232 5039 scope.go:122] "RemoveContainer" containerID="735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c" Oct 30 22:12:15.764628 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.764614 5039 scope.go:122] "RemoveContainer" containerID="735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c" Oct 30 22:12:15.764945 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:12:15.764927 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c\": container with ID starting with 735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c not found: ID does not exist" containerID="735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c" Oct 30 22:12:15.765010 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.764956 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c"} err="failed to get container status \"735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c\": rpc error: code = NotFound desc = could not find container \"735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c\": container with ID starting with 735c4ebb2f1e23733c1e98dab095f7beed82748d89a46ed54522d8a390b0092c not found: ID does not exist" Oct 30 22:12:15.813499 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.809734 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933a2d98-249d-4af7-a3f7-a73b8303410c-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:15.813499 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.809769 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:15.813499 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.809781 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933a2d98-249d-4af7-a3f7-a73b8303410c-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:15.813499 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.809794 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkgc\" (UniqueName: \"kubernetes.io/projected/933a2d98-249d-4af7-a3f7-a73b8303410c-kube-api-access-clkgc\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:15.817210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.817173 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn"] Oct 30 22:12:15.833575 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:15.833538 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5454af/test-model-5454af-kserve-cb7c5848f-22mtn"] Oct 30 22:12:16.268509 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:16.268474 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:16.444330 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:16.444293 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:16.782512 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:16.782477 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v"] Oct 30 22:12:16.889411 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:16.883924 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:16.929084 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:16.929047 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933a2d98-249d-4af7-a3f7-a73b8303410c" path="/var/lib/kubelet/pods/933a2d98-249d-4af7-a3f7-a73b8303410c/volumes" Oct 30 22:12:17.064686 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:17.064601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:17.287413 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:17.287289 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:17.974323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:17.974290 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:18.346594 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:18.346496 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:20.210833 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:20.210799 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:21.863358 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:21.863328 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.302160 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:23.302128 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.460898 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:23.460860 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.615225 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:23.615145 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.774052 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:23.774006 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:25.183503 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:12:25.183466 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:12:25.379061 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:25.379022 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:27.033975 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:27.033945 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:27.720546 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:27.720508 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:27.871320 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:27.871272 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:28.929072 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:28.929037 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:30.577876 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:30.577831 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:31.803543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:31.803501 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" podUID="94a805cf-6f02-4e24-9205-1f729fc615fa" containerName="main" containerID="cri-o://ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023" gracePeriod=45 Oct 30 22:12:32.072971 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.072949 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:12:32.196465 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196435 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm\") pod \"94a805cf-6f02-4e24-9205-1f729fc615fa\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " Oct 30 22:12:32.196620 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196469 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs\") pod \"94a805cf-6f02-4e24-9205-1f729fc615fa\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " Oct 30 22:12:32.196620 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196496 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6\" (UniqueName: \"kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6\") pod \"94a805cf-6f02-4e24-9205-1f729fc615fa\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " Oct 30 22:12:32.196620 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196536 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache\") pod \"94a805cf-6f02-4e24-9205-1f729fc615fa\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " Oct 30 22:12:32.196620 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196575 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir\") pod \"94a805cf-6f02-4e24-9205-1f729fc615fa\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " Oct 30 22:12:32.196834 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196631 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home\") pod \"94a805cf-6f02-4e24-9205-1f729fc615fa\" (UID: \"94a805cf-6f02-4e24-9205-1f729fc615fa\") " Oct 30 22:12:32.196834 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196786 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache" pod "94a805cf-6f02-4e24-9205-1f729fc615fa" (UID: "94a805cf-6f02-4e24-9205-1f729fc615fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:32.196940 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196895 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir" pod "94a805cf-6f02-4e24-9205-1f729fc615fa" (UID: "94a805cf-6f02-4e24-9205-1f729fc615fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:32.196940 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.196922 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home" pod "94a805cf-6f02-4e24-9205-1f729fc615fa" (UID: "94a805cf-6f02-4e24-9205-1f729fc615fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:32.197167 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.197149 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:32.197238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.197172 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:32.197238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.197181 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:32.198648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.198628 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm" pod "94a805cf-6f02-4e24-9205-1f729fc615fa" (UID: "94a805cf-6f02-4e24-9205-1f729fc615fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:32.199018 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.198995 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs" pod "94a805cf-6f02-4e24-9205-1f729fc615fa" (UID: "94a805cf-6f02-4e24-9205-1f729fc615fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:32.199106 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.199028 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6" pod "94a805cf-6f02-4e24-9205-1f729fc615fa" (UID: "94a805cf-6f02-4e24-9205-1f729fc615fa"). InnerVolumeSpecName "kube-api-access-7frp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:32.298204 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.298184 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94a805cf-6f02-4e24-9205-1f729fc615fa-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:32.298327 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.298205 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94a805cf-6f02-4e24-9205-1f729fc615fa-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:32.298327 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.298216 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frp6\" (UniqueName: \"kubernetes.io/projected/94a805cf-6f02-4e24-9205-1f729fc615fa-kube-api-access-7frp6\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:32.321409 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.321310 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:32.808021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.807979 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:32.877313 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.877279 5039 generic.go:356] "Generic (PLEG): container finished" podID="94a805cf-6f02-4e24-9205-1f729fc615fa" containerID="ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023" exitCode=0 Oct 30 22:12:32.877548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.877424 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" Oct 30 22:12:32.877548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.877447 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" event={"ID":"94a805cf-6f02-4e24-9205-1f729fc615fa","Type":"ContainerDied","Data":"ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023"} Oct 30 22:12:32.877548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.877483 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v" event={"ID":"94a805cf-6f02-4e24-9205-1f729fc615fa","Type":"ContainerDied","Data":"a7ab14c3bd4aefa21a28ab47e7af3ad3e6532d0844977c4c30b633621feac3df"} Oct 30 22:12:32.877548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.877504 5039 scope.go:122] "RemoveContainer" containerID="ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023" Oct 30 22:12:32.903757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.903739 5039 scope.go:122] "RemoveContainer" containerID="ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023" Oct 30 22:12:32.904893 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:12:32.904869 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023\": container with ID starting with ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023 not found: ID does not exist" containerID="ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023" Oct 30 22:12:32.904994 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.904899 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023"} err="failed to get container status \"ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023\": rpc error: code = NotFound desc = could not find container \"ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023\": container with ID starting with ed2ac3b5e7b4b8d709560e9cbb1b9af6894a41c5e5884117bd023e8ecfc28023 not found: ID does not exist" Oct 30 22:12:32.923771 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.923745 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v"] Oct 30 22:12:32.933215 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:32.933191 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-589ebc/test-model-589ebc-kserve-849cc9fcc5-8mh7v"] Oct 30 22:12:34.231358 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:34.231323 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:34.853676 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:34.853625 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:34.921735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:34.921701 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a805cf-6f02-4e24-9205-1f729fc615fa" path="/var/lib/kubelet/pods/94a805cf-6f02-4e24-9205-1f729fc615fa/volumes" Oct 30 22:12:35.007511 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:35.007478 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:35.913570 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:35.913521 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:37.561301 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:37.561267 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:39.297251 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:39.297205 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:40.923756 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:40.922672 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:42.602739 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:42.602694 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:43.670863 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:43.670829 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:43.812475 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:43.812440 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:44.234157 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:44.234120 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:45.531895 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.531867 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h"] Oct 30 22:12:45.532670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.532059 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" podUID="4d128504-5584-4533-9f19-1c00d1c10c76" containerName="maas-api" containerID="cri-o://c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2" gracePeriod=30 Oct 30 22:12:45.647978 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.647950 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:45.830642 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.830608 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 22:12:45.957331 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.957300 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:12:45.964867 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.964841 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p\" (UniqueName: \"kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p\") pod \"4d128504-5584-4533-9f19-1c00d1c10c76\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " Oct 30 22:12:45.965019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.964875 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs\") pod \"4d128504-5584-4533-9f19-1c00d1c10c76\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " Oct 30 22:12:45.965019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.964906 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca\") pod \"4d128504-5584-4533-9f19-1c00d1c10c76\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " Oct 30 22:12:45.965110 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.965033 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\") pod \"4d128504-5584-4533-9f19-1c00d1c10c76\" (UID: \"4d128504-5584-4533-9f19-1c00d1c10c76\") " Oct 30 22:12:45.965471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.965433 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca" pod "4d128504-5584-4533-9f19-1c00d1c10c76" (UID: "4d128504-5584-4533-9f19-1c00d1c10c76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:45.966598 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.966571 5039 generic.go:356] "Generic (PLEG): container finished" podID="4d128504-5584-4533-9f19-1c00d1c10c76" containerID="c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2" exitCode=0 Oct 30 22:12:45.966725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.966617 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" Oct 30 22:12:45.966725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.966625 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" event={"ID":"4d128504-5584-4533-9f19-1c00d1c10c76","Type":"ContainerDied","Data":"c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2"} Oct 30 22:12:45.966725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.966675 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h" event={"ID":"4d128504-5584-4533-9f19-1c00d1c10c76","Type":"ContainerDied","Data":"f3197ff5c598bbb5291567acd4dc62822dc418cab522b20b5600593229683262"} Oct 30 22:12:45.966725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.966703 5039 scope.go:122] "RemoveContainer" containerID="c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2" Oct 30 22:12:45.967429 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.967381 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls" pod "4d128504-5584-4533-9f19-1c00d1c10c76" (UID: "4d128504-5584-4533-9f19-1c00d1c10c76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:45.967639 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.967614 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p" pod "4d128504-5584-4533-9f19-1c00d1c10c76" (UID: "4d128504-5584-4533-9f19-1c00d1c10c76"). InnerVolumeSpecName "kube-api-access-bzj7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:45.967741 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.967615 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs" pod "4d128504-5584-4533-9f19-1c00d1c10c76" (UID: "4d128504-5584-4533-9f19-1c00d1c10c76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:45.985738 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.985721 5039 scope.go:122] "RemoveContainer" containerID="c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2" Oct 30 22:12:45.986023 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:12:45.986003 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2\": container with ID starting with c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2 not found: ID does not exist" containerID="c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2" Oct 30 22:12:45.986081 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:45.986035 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2"} err="failed to get container status \"c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2\": rpc error: code = NotFound desc = could not find container \"c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2\": container with ID starting with c47be08a649ae5b6a51ba6933b3b249b0172fb7a45129377ab5d190a73d5a7d2 not found: ID does not exist" Oct 30 22:12:46.066503 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.066413 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzj7p\" (UniqueName: \"kubernetes.io/projected/4d128504-5584-4533-9f19-1c00d1c10c76-kube-api-access-bzj7p\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:46.066503 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.066449 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:46.066503 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.066467 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d128504-5584-4533-9f19-1c00d1c10c76-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:46.066503 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.066480 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d128504-5584-4533-9f19-1c00d1c10c76-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:12:46.216346 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.216261 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.319596 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.319510 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h"] Oct 30 22:12:46.329260 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.329231 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c19461-5bb7f96bc8-mj28h"] Oct 30 22:12:46.419760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.419729 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.857310 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.857257 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.921145 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:46.921115 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d128504-5584-4533-9f19-1c00d1c10c76" path="/var/lib/kubelet/pods/4d128504-5584-4533-9f19-1c00d1c10c76/volumes" Oct 30 22:12:47.981689 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:47.981654 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:48.669705 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:48.669672 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:48.872263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:48.872225 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.285596 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:49.285565 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:50.313974 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:50.313931 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:50.885891 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:50.885855 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:51.056090 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:51.056055 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:51.490586 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:51.490553 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:52.620318 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:52.620283 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:53.570641 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:53.570602 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:53.723770 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:53.723730 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:53.987792 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:53.967736 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:54.411002 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:54.410964 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:55.184021 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:12:55.183989 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:12:55.683846 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:55.683802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:55.854212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:55.854174 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:55.961564 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:55.961486 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk"] Oct 30 22:12:55.961752 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:55.961722 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" containerID="cri-o://850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8" gracePeriod=30 Oct 30 22:12:56.044177 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:56.044147 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:56.192641 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:56.192601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:57.759036 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:57.759002 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:57.908489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:57.908455 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:58.077954 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:58.077868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:58.219115 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:58.219083 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:59.805902 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:12:59.805861 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:01.227745 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.227718 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:13:01.237723 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237674 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.237723 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237710 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.237859 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237752 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.237859 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237800 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.237859 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237844 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.238001 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237879 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.238001 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237897 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.238001 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237921 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:01.238001 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.237926 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.238236 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238213 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:01.238312 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238155 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:01.238440 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238418 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:01.238590 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238570 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.238670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238607 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.238670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238625 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.238670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.238640 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.239817 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.239782 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:01.239938 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.239865 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "kube-api-access-hsh5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:01.240126 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.240100 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:01.240240 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.240177 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:01.339159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339128 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs\") pod \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\" (UID: \"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c\") " Oct 30 22:13:01.339353 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339334 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs" pod "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" (UID: "2ff95999-64cd-49b1-aa4a-cf11bb3eb20c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:01.339609 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339590 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.339680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339614 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.339680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339627 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsh5s\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-kube-api-access-hsh5s\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.339680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339642 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.339680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.339656 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:01.534705 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:01.534626 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:02.066650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.066617 5039 generic.go:356] "Generic (PLEG): container finished" podID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerID="850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8" exitCode=0 Oct 30 22:13:02.066814 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.066693 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" Oct 30 22:13:02.066814 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.066694 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" event={"ID":"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c","Type":"ContainerDied","Data":"850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8"} Oct 30 22:13:02.066814 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.066746 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk" event={"ID":"2ff95999-64cd-49b1-aa4a-cf11bb3eb20c","Type":"ContainerDied","Data":"233c6996c394da20fce4535066077654710aad9ab1fa6b09ee7d1b0973c1b8cd"} Oct 30 22:13:02.066814 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.066767 5039 scope.go:122] "RemoveContainer" containerID="850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8" Oct 30 22:13:02.080255 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.080104 5039 scope.go:122] "RemoveContainer" containerID="13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150" Oct 30 22:13:02.089382 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.089367 5039 scope.go:122] "RemoveContainer" containerID="850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8" Oct 30 22:13:02.089657 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:13:02.089636 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8\": container with ID starting with 850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8 not found: ID does not exist" containerID="850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8" Oct 30 22:13:02.089733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.089672 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8"} err="failed to get container status \"850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8\": rpc error: code = NotFound desc = could not find container \"850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8\": container with ID starting with 850faf5d1c7a6d45eb25282fc5ea4f665a84f1cd1575fa185d6bb0918f8d88c8 not found: ID does not exist" Oct 30 22:13:02.089733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.089694 5039 scope.go:122] "RemoveContainer" containerID="13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150" Oct 30 22:13:02.089943 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:13:02.089923 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150\": container with ID starting with 13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150 not found: ID does not exist" containerID="13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150" Oct 30 22:13:02.089980 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.089950 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150"} err="failed to get container status \"13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150\": rpc error: code = NotFound desc = could not find container \"13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150\": container with ID starting with 13ef41d8153456840ec488d3475325f5e30547212c6a7617c17ccef0219e6150 not found: ID does not exist" Oct 30 22:13:02.121808 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.121786 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk"] Oct 30 22:13:02.132866 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.132842 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a42c31-openshift-default-5cfb8dc8ff-wnbvk"] Oct 30 22:13:02.921644 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:02.921575 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" path="/var/lib/kubelet/pods/2ff95999-64cd-49b1-aa4a-cf11bb3eb20c/volumes" Oct 30 22:13:03.251500 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:03.251408 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:04.914813 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:04.914778 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:05.077056 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:05.077017 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:05.219460 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:05.219365 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:06.543967 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:06.543927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:07.164698 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:07.164670 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:07.328015 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:07.327986 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:08.076424 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:08.076361 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:09.220206 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:09.220167 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:09.369213 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:09.369174 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:09.698312 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:09.698276 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:11.304101 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:11.304067 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:12.980783 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:12.980753 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:14.732075 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:14.732041 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:16.422013 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:16.421967 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:16.824184 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:16.824111 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:13:18.006659 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:18.006615 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:18.153949 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:18.153914 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:19.565414 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:19.562331 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:20.807873 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:20.807840 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz"] Oct 30 22:13:20.808343 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:20.808108 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" podUID="dcb6bb80-25ca-4984-82af-293b8f254cd3" containerName="maas-api" containerID="cri-o://39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9" gracePeriod=30 Oct 30 22:13:20.929560 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:20.929465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.121559 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.121536 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 22:13:21.182340 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.181260 5039 generic.go:356] "Generic (PLEG): container finished" podID="dcb6bb80-25ca-4984-82af-293b8f254cd3" containerID="39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9" exitCode=0 Oct 30 22:13:21.182340 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.181501 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" event={"ID":"dcb6bb80-25ca-4984-82af-293b8f254cd3","Type":"ContainerDied","Data":"39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9"} Oct 30 22:13:21.182340 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.181546 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" event={"ID":"dcb6bb80-25ca-4984-82af-293b8f254cd3","Type":"ContainerDied","Data":"397736fc4683781224b99707690339a3f4d083f3875cbdf1e1176ed1c3a850fd"} Oct 30 22:13:21.182340 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.181579 5039 scope.go:122] "RemoveContainer" containerID="39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9" Oct 30 22:13:21.182340 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.181736 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz" Oct 30 22:13:21.195817 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.195784 5039 scope.go:122] "RemoveContainer" containerID="39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9" Oct 30 22:13:21.196287 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:13:21.196260 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9\": container with ID starting with 39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9 not found: ID does not exist" containerID="39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9" Oct 30 22:13:21.196425 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.196301 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9"} err="failed to get container status \"39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9\": rpc error: code = NotFound desc = could not find container \"39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9\": container with ID starting with 39b3275562d45bd74597c698223f8466e697b80419e3d2ca1a68d3c69f32f4b9 not found: ID does not exist" Oct 30 22:13:21.237299 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.237265 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca\") pod \"dcb6bb80-25ca-4984-82af-293b8f254cd3\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " Oct 30 22:13:21.237507 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.237328 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs\") pod \"dcb6bb80-25ca-4984-82af-293b8f254cd3\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " Oct 30 22:13:21.237507 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.237458 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2\" (UniqueName: \"kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2\") pod \"dcb6bb80-25ca-4984-82af-293b8f254cd3\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " Oct 30 22:13:21.237637 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.237546 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls\") pod \"dcb6bb80-25ca-4984-82af-293b8f254cd3\" (UID: \"dcb6bb80-25ca-4984-82af-293b8f254cd3\") " Oct 30 22:13:21.237694 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.237670 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca" pod "dcb6bb80-25ca-4984-82af-293b8f254cd3" (UID: "dcb6bb80-25ca-4984-82af-293b8f254cd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:21.239592 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.238171 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb6bb80-25ca-4984-82af-293b8f254cd3-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:21.239884 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.239857 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs" pod "dcb6bb80-25ca-4984-82af-293b8f254cd3" (UID: "dcb6bb80-25ca-4984-82af-293b8f254cd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:21.239982 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.239895 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2" pod "dcb6bb80-25ca-4984-82af-293b8f254cd3" (UID: "dcb6bb80-25ca-4984-82af-293b8f254cd3"). InnerVolumeSpecName "kube-api-access-sxbq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:21.240321 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.240301 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls" pod "dcb6bb80-25ca-4984-82af-293b8f254cd3" (UID: "dcb6bb80-25ca-4984-82af-293b8f254cd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:21.296808 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.296776 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.339753 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.339726 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:21.339861 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.339759 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbq2\" (UniqueName: \"kubernetes.io/projected/dcb6bb80-25ca-4984-82af-293b8f254cd3-kube-api-access-sxbq2\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:21.339861 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.339774 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcb6bb80-25ca-4984-82af-293b8f254cd3-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:21.478916 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.478864 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.530240 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.530215 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz"] Oct 30 22:13:21.544197 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.544174 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846656-796845fcdb-gffpz"] Oct 30 22:13:21.563693 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.563663 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb"] Oct 30 22:13:21.563875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.563852 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" podUID="cb181f48-bb52-4eff-8a23-d792758d7f53" containerName="maas-api" containerID="cri-o://5c426cb2b1d309be5a89f759ef4ca2c656cf40e76cd7d884128a4f538ee60ff0" gracePeriod=30 Oct 30 22:13:21.635429 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.633738 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.793728 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.793690 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.844977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.844955 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:13:21.946407 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.946369 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls\") pod \"cb181f48-bb52-4eff-8a23-d792758d7f53\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " Oct 30 22:13:21.946538 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.946465 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs\") pod \"cb181f48-bb52-4eff-8a23-d792758d7f53\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " Oct 30 22:13:21.946538 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.946487 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4\" (UniqueName: \"kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4\") pod \"cb181f48-bb52-4eff-8a23-d792758d7f53\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " Oct 30 22:13:21.946538 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.946514 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca\") pod \"cb181f48-bb52-4eff-8a23-d792758d7f53\" (UID: \"cb181f48-bb52-4eff-8a23-d792758d7f53\") " Oct 30 22:13:21.946936 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.946913 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca" pod "cb181f48-bb52-4eff-8a23-d792758d7f53" (UID: "cb181f48-bb52-4eff-8a23-d792758d7f53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:21.948464 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.948432 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls" pod "cb181f48-bb52-4eff-8a23-d792758d7f53" (UID: "cb181f48-bb52-4eff-8a23-d792758d7f53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:21.948562 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.948461 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs" pod "cb181f48-bb52-4eff-8a23-d792758d7f53" (UID: "cb181f48-bb52-4eff-8a23-d792758d7f53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:21.948562 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:21.948463 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4" pod "cb181f48-bb52-4eff-8a23-d792758d7f53" (UID: "cb181f48-bb52-4eff-8a23-d792758d7f53"). InnerVolumeSpecName "kube-api-access-fhvv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:22.075471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.055096 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.075471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.059678 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:22.075471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.059703 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb181f48-bb52-4eff-8a23-d792758d7f53-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:22.075471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.059717 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvv4\" (UniqueName: \"kubernetes.io/projected/cb181f48-bb52-4eff-8a23-d792758d7f53-kube-api-access-fhvv4\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:22.075471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.059731 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb181f48-bb52-4eff-8a23-d792758d7f53-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:22.173192 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.173165 5039 scope.go:122] "RemoveContainer" containerID="5c426cb2b1d309be5a89f759ef4ca2c656cf40e76cd7d884128a4f538ee60ff0" Oct 30 22:13:22.188172 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.188154 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" Oct 30 22:13:22.188247 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.188187 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" event={"ID":"cb181f48-bb52-4eff-8a23-d792758d7f53","Type":"ContainerDied","Data":"5c426cb2b1d309be5a89f759ef4ca2c656cf40e76cd7d884128a4f538ee60ff0"} Oct 30 22:13:22.188247 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.188236 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb" event={"ID":"cb181f48-bb52-4eff-8a23-d792758d7f53","Type":"ContainerDied","Data":"1e8d1ed26556c74561b32cf505a358b4bda9042185c717fa93fb19df8855d7c0"} Oct 30 22:13:22.199711 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.199675 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.232899 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.232878 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb"] Oct 30 22:13:22.242568 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.242547 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19a725-6976dfd4bb-b9zwb"] Oct 30 22:13:22.352268 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.352219 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.498988 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.498950 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.848854 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.848819 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.921420 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.921376 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb181f48-bb52-4eff-8a23-d792758d7f53" path="/var/lib/kubelet/pods/cb181f48-bb52-4eff-8a23-d792758d7f53/volumes" Oct 30 22:13:22.921798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:22.921783 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb6bb80-25ca-4984-82af-293b8f254cd3" path="/var/lib/kubelet/pods/dcb6bb80-25ca-4984-82af-293b8f254cd3/volumes" Oct 30 22:13:24.185535 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:24.185492 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:25.183148 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:13:25.183114 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:13:25.929975 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:25.929937 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:27.632160 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:27.632116 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:29.408541 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:29.408508 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:30.793265 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:30.793231 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:31.091839 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:31.091756 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j"] Oct 30 22:13:31.092022 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:31.091972 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" containerID="cri-o://ef4f55fac712e9c8f6f948e77588c9671d6c7348940a9a37a49c7433f0b814af" gracePeriod=30 Oct 30 22:13:31.096734 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:31.096705 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:31.352284 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:31.352204 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:31.605449 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:31.605346 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:32.158452 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:32.158420 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:32.376769 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:32.376730 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:32.552663 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:32.552578 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:33.647793 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:33.647731 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:35.410910 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:35.410866 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:36.280419 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.279382 5039 generic.go:356] "Generic (PLEG): container finished" podID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerID="ef4f55fac712e9c8f6f948e77588c9671d6c7348940a9a37a49c7433f0b814af" exitCode=0 Oct 30 22:13:36.280419 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.279445 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" event={"ID":"e8a56bfa-062d-4b01-9778-8d9954e1d517","Type":"ContainerDied","Data":"ef4f55fac712e9c8f6f948e77588c9671d6c7348940a9a37a49c7433f0b814af"} Oct 30 22:13:36.341508 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.341482 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 22:13:36.399947 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.399923 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400104 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.399986 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400104 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400024 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400104 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400060 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400273 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400127 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400273 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400166 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400375 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:36.400518 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400467 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400544 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400626 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket\") pod \"e8a56bfa-062d-4b01-9778-8d9954e1d517\" (UID: \"e8a56bfa-062d-4b01-9778-8d9954e1d517\") " Oct 30 22:13:36.400855 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.400834 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:36.401033 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.401014 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:36.401191 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.401173 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:36.401259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.401197 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.401259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.401219 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a56bfa-062d-4b01-9778-8d9954e1d517-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.401259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.401237 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:36.401259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.401234 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.403203 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.403175 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:36.403319 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.403182 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:36.403544 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.403508 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:36.403650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.403594 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc" pod "e8a56bfa-062d-4b01-9778-8d9954e1d517" (UID: "e8a56bfa-062d-4b01-9778-8d9954e1d517"). InnerVolumeSpecName "kube-api-access-64kpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:36.502455 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.502431 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.502455 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.502455 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.502767 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.502466 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a56bfa-062d-4b01-9778-8d9954e1d517-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.502767 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.502475 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64kpc\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-kube-api-access-64kpc\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.502767 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.502485 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:36.502767 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:36.502493 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a56bfa-062d-4b01-9778-8d9954e1d517-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:37.016304 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:37.016266 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:37.287084 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:37.287017 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" Oct 30 22:13:37.287221 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:37.287016 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j" event={"ID":"e8a56bfa-062d-4b01-9778-8d9954e1d517","Type":"ContainerDied","Data":"2231a4dcf6a5c1cffcbb8b7992718a0362804e4da6dba5d190035cfb6621961a"} Oct 30 22:13:37.287221 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:37.287145 5039 scope.go:122] "RemoveContainer" containerID="ef4f55fac712e9c8f6f948e77588c9671d6c7348940a9a37a49c7433f0b814af" Oct 30 22:13:37.344425 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:37.344385 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j"] Oct 30 22:13:37.353724 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:37.353701 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lm68j"] Oct 30 22:13:38.668355 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:38.668319 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:38.921573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:38.921501 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" path="/var/lib/kubelet/pods/e8a56bfa-062d-4b01-9778-8d9954e1d517/volumes" Oct 30 22:13:40.320166 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:40.320113 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:41.891919 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:41.891884 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:42.331014 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:42.330940 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:42.341166 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:42.341140 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:13:42.490652 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:42.490602 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:43.127826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:43.127786 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:43.297629 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:43.297591 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:43.592935 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:43.592877 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:43.936257 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:43.936226 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:44.103742 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:44.103708 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:45.267138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:45.267097 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:46.995779 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:46.995739 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.460531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:48.460491 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.819289 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:48.819209 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.988579 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:48.988542 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.162555 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:49.162514 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.536247 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:49.536173 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.702996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:49.702958 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:50.143849 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:50.143812 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.421146 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:51.421112 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.098465 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:53.098350 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:54.801369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:54.801336 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:55.183686 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:13:55.183654 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:13:56.083382 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.083343 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz"] Oct 30 22:13:56.083889 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.083619 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" podUID="ea9761f2-8a71-4202-881a-13c7572defdd" containerName="maas-api" containerID="cri-o://52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638" gracePeriod=30 Oct 30 22:13:56.204022 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.203992 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.382497 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.382476 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 22:13:56.393844 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.393821 5039 generic.go:356] "Generic (PLEG): container finished" podID="ea9761f2-8a71-4202-881a-13c7572defdd" containerID="52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638" exitCode=0 Oct 30 22:13:56.393957 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.393856 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" Oct 30 22:13:56.393957 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.393901 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" event={"ID":"ea9761f2-8a71-4202-881a-13c7572defdd","Type":"ContainerDied","Data":"52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638"} Oct 30 22:13:56.393957 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.393946 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz" event={"ID":"ea9761f2-8a71-4202-881a-13c7572defdd","Type":"ContainerDied","Data":"ead51785cfbaf34c3419d63055408a00d76f534d9fa05343113841766c036eda"} Oct 30 22:13:56.394058 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.393967 5039 scope.go:122] "RemoveContainer" containerID="52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638" Oct 30 22:13:56.403064 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.403045 5039 scope.go:122] "RemoveContainer" containerID="52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638" Oct 30 22:13:56.403345 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:13:56.403327 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638\": container with ID starting with 52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638 not found: ID does not exist" containerID="52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638" Oct 30 22:13:56.403415 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.403356 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638"} err="failed to get container status \"52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638\": rpc error: code = NotFound desc = could not find container \"52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638\": container with ID starting with 52eabf777883b7eeaded4c4cf748d31e4e8fe16933f628058f729129ec1a5638 not found: ID does not exist" Oct 30 22:13:56.507766 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.507743 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs\") pod \"ea9761f2-8a71-4202-881a-13c7572defdd\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " Oct 30 22:13:56.507891 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.507879 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca\") pod \"ea9761f2-8a71-4202-881a-13c7572defdd\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " Oct 30 22:13:56.507933 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.507912 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8\") pod \"ea9761f2-8a71-4202-881a-13c7572defdd\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " Oct 30 22:13:56.507977 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.507960 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls\") pod \"ea9761f2-8a71-4202-881a-13c7572defdd\" (UID: \"ea9761f2-8a71-4202-881a-13c7572defdd\") " Oct 30 22:13:56.508264 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.508238 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca" pod "ea9761f2-8a71-4202-881a-13c7572defdd" (UID: "ea9761f2-8a71-4202-881a-13c7572defdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:56.509826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.509801 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs" pod "ea9761f2-8a71-4202-881a-13c7572defdd" (UID: "ea9761f2-8a71-4202-881a-13c7572defdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:56.509922 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.509906 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls" pod "ea9761f2-8a71-4202-881a-13c7572defdd" (UID: "ea9761f2-8a71-4202-881a-13c7572defdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:56.510023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.510007 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8" pod "ea9761f2-8a71-4202-881a-13c7572defdd" (UID: "ea9761f2-8a71-4202-881a-13c7572defdd"). InnerVolumeSpecName "kube-api-access-rqkn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:56.609097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.609013 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9761f2-8a71-4202-881a-13c7572defdd-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:56.609097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.609044 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/ea9761f2-8a71-4202-881a-13c7572defdd-kube-api-access-rqkn8\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:56.609097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.609053 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:56.609097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.609062 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea9761f2-8a71-4202-881a-13c7572defdd-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:13:56.748498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.748469 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz"] Oct 30 22:13:56.784343 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.784313 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-09eabe-8bdf98f7f-tqxjz"] Oct 30 22:13:56.811726 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.811701 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.921489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.921466 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9761f2-8a71-4202-881a-13c7572defdd" path="/var/lib/kubelet/pods/ea9761f2-8a71-4202-881a-13c7572defdd/volumes" Oct 30 22:13:56.996983 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:56.996960 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:57.154757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.154721 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:57.262241 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.262167 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l"] Oct 30 22:13:57.262467 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.262441 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" containerID="cri-o://11ff86d0a16aa8cb0ece0ca34603ae3574baeb85f18b694b46408adc2dd3de65" gracePeriod=30 Oct 30 22:13:57.386438 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.385843 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:57.448788 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.448756 5039 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:57.448954 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.448809 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:57.615209 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:57.615140 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:58.136439 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:58.136371 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:58.357857 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:58.357825 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:58.533404 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:58.533296 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:58.917086 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:58.917040 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:59.367758 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:13:59.367681 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:14:00.705897 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:00.705852 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:02.434725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.434686 5039 generic.go:356] "Generic (PLEG): container finished" podID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerID="11ff86d0a16aa8cb0ece0ca34603ae3574baeb85f18b694b46408adc2dd3de65" exitCode=0 Oct 30 22:14:02.435090 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.434757 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" event={"ID":"3b968a90-d6dd-481f-b845-8bc0463ffae5","Type":"ContainerDied","Data":"11ff86d0a16aa8cb0ece0ca34603ae3574baeb85f18b694b46408adc2dd3de65"} Oct 30 22:14:02.435090 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.434840 5039 scope.go:122] "RemoveContainer" containerID="d3edc710d828e9b2e69656ca273591b91a18687a412d23ec76bcf27bd0512647" Oct 30 22:14:02.470192 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.470084 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:02.525624 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.525600 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 22:14:02.679450 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679427 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679479 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679516 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679548 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679584 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679624 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679683 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:02.679735 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679714 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679742 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679787 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy\") pod \"3b968a90-d6dd-481f-b845-8bc0463ffae5\" (UID: \"3b968a90-d6dd-481f-b845-8bc0463ffae5\") " Oct 30 22:14:02.679930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679833 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:02.679930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679849 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:02.680119 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679929 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:02.680119 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.679943 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:02.680279 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.680256 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.680337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.680286 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.680337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.680303 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.680337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.680320 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.680337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.680333 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b968a90-d6dd-481f-b845-8bc0463ffae5-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.681779 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.681758 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:02.681865 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.681796 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:02.681865 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.681817 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:02.681943 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.681881 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm" pod "3b968a90-d6dd-481f-b845-8bc0463ffae5" (UID: "3b968a90-d6dd-481f-b845-8bc0463ffae5"). InnerVolumeSpecName "kube-api-access-n6pmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:02.781077 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.781054 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.781077 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.781077 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.781198 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.781086 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6pmm\" (UniqueName: \"kubernetes.io/projected/3b968a90-d6dd-481f-b845-8bc0463ffae5-kube-api-access-n6pmm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:02.781198 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:02.781095 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b968a90-d6dd-481f-b845-8bc0463ffae5-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:03.442255 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:03.442219 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" event={"ID":"3b968a90-d6dd-481f-b845-8bc0463ffae5","Type":"ContainerDied","Data":"f79197ada53f1cc70f740db13db4f827848c1f5b4a128b4370fa7a686525547b"} Oct 30 22:14:03.442255 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:03.442235 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l" Oct 30 22:14:03.442746 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:03.442283 5039 scope.go:122] "RemoveContainer" containerID="11ff86d0a16aa8cb0ece0ca34603ae3574baeb85f18b694b46408adc2dd3de65" Oct 30 22:14:03.497071 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:03.497048 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l"] Oct 30 22:14:03.507102 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:03.507076 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-ms44l"] Oct 30 22:14:04.005514 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:04.005473 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:04.287151 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:04.287073 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:04.532716 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:04.532682 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:04.703961 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:04.703927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:04.920710 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:04.920679 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" path="/var/lib/kubelet/pods/3b968a90-d6dd-481f-b845-8bc0463ffae5/volumes" Oct 30 22:14:05.602514 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:05.602481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:05.765332 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:05.765296 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:05.930139 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:05.930109 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:07.681644 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:07.681585 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:08.507554 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:08.507516 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:08.650960 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:08.650927 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.542174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:09.542137 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.701437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:09.701381 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.857725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:09.857649 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:10.985732 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:10.985692 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp"] Oct 30 22:14:10.986144 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:10.985933 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" podUID="bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" containerName="maas-api" containerID="cri-o://b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9" gracePeriod=30 Oct 30 22:14:11.083487 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.083453 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:11.264504 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.264478 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 22:14:11.341861 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.341834 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk"] Oct 30 22:14:11.342053 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.342028 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" podUID="aa6ff71c-06fb-4546-a65e-d0ac053f8637" containerName="maas-api" containerID="cri-o://3f82f170f4d14ded5ca998ded9700b01474d0844a4a950914fe12d1946bb1c78" gracePeriod=30 Oct 30 22:14:11.364308 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.364279 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs\") pod \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " Oct 30 22:14:11.364441 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.364366 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca\") pod \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " Oct 30 22:14:11.364516 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.364438 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls\") pod \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " Oct 30 22:14:11.364587 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.364515 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv\" (UniqueName: \"kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv\") pod \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\" (UID: \"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6\") " Oct 30 22:14:11.364954 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.364926 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca" pod "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" (UID: "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:11.369754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.369407 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs" pod "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" (UID: "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:11.369754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.369427 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls" pod "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" (UID: "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:11.369754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.369453 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv" pod "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" (UID: "bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6"). InnerVolumeSpecName "kube-api-access-wfsnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:11.480241 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.480212 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.480241 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.480242 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.480471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.480252 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.480471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.480261 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfsnv\" (UniqueName: \"kubernetes.io/projected/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6-kube-api-access-wfsnv\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.481075 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.481048 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:11.492774 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.492702 5039 generic.go:356] "Generic (PLEG): container finished" podID="aa6ff71c-06fb-4546-a65e-d0ac053f8637" containerID="3f82f170f4d14ded5ca998ded9700b01474d0844a4a950914fe12d1946bb1c78" exitCode=0 Oct 30 22:14:11.492896 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.492784 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" event={"ID":"aa6ff71c-06fb-4546-a65e-d0ac053f8637","Type":"ContainerDied","Data":"3f82f170f4d14ded5ca998ded9700b01474d0844a4a950914fe12d1946bb1c78"} Oct 30 22:14:11.494536 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.494497 5039 generic.go:356] "Generic (PLEG): container finished" podID="bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" containerID="b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9" exitCode=0 Oct 30 22:14:11.494666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.494552 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" event={"ID":"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6","Type":"ContainerDied","Data":"b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9"} Oct 30 22:14:11.494666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.494604 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" event={"ID":"bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6","Type":"ContainerDied","Data":"1ec3cd7195485a697a0ed962e19cd526a8df4c613dfe3a7dbbef135007ff6db6"} Oct 30 22:14:11.494666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.494624 5039 scope.go:122] "RemoveContainer" containerID="b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9" Oct 30 22:14:11.494666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.494561 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp" Oct 30 22:14:11.514765 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.514748 5039 scope.go:122] "RemoveContainer" containerID="b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9" Oct 30 22:14:11.515073 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:11.515045 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9\": container with ID starting with b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9 not found: ID does not exist" containerID="b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9" Oct 30 22:14:11.515155 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.515084 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9"} err="failed to get container status \"b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9\": rpc error: code = NotFound desc = could not find container \"b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9\": container with ID starting with b498072b0e3b8656c529cb906378454f7072897cbadebf1309d38af6bc49b9e9 not found: ID does not exist" Oct 30 22:14:11.549554 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.549526 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp"] Oct 30 22:14:11.552442 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.552417 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9722ac-76989dc565-xl6cp"] Oct 30 22:14:11.608471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.608450 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 22:14:11.670985 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.670955 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:11.681960 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.681936 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca\") pod \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " Oct 30 22:14:11.682091 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.681991 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs\") pod \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " Oct 30 22:14:11.682091 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.682081 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\") pod \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " Oct 30 22:14:11.682222 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.682208 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g\" (UniqueName: \"kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g\") pod \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\" (UID: \"aa6ff71c-06fb-4546-a65e-d0ac053f8637\") " Oct 30 22:14:11.682320 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.682295 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca" pod "aa6ff71c-06fb-4546-a65e-d0ac053f8637" (UID: "aa6ff71c-06fb-4546-a65e-d0ac053f8637"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:11.682594 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.682572 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6ff71c-06fb-4546-a65e-d0ac053f8637-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.684174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.684146 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls" pod "aa6ff71c-06fb-4546-a65e-d0ac053f8637" (UID: "aa6ff71c-06fb-4546-a65e-d0ac053f8637"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:11.684174 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.684163 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs" pod "aa6ff71c-06fb-4546-a65e-d0ac053f8637" (UID: "aa6ff71c-06fb-4546-a65e-d0ac053f8637"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:11.684331 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.684239 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g" pod "aa6ff71c-06fb-4546-a65e-d0ac053f8637" (UID: "aa6ff71c-06fb-4546-a65e-d0ac053f8637"). InnerVolumeSpecName "kube-api-access-6gp4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:11.805093 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.804998 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.805093 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.805035 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gp4g\" (UniqueName: \"kubernetes.io/projected/aa6ff71c-06fb-4546-a65e-d0ac053f8637-kube-api-access-6gp4g\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.805093 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.805049 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa6ff71c-06fb-4546-a65e-d0ac053f8637-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:11.832998 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:11.832959 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.109331 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.109265 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.264835 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.264797 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.502960 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.502931 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" event={"ID":"aa6ff71c-06fb-4546-a65e-d0ac053f8637","Type":"ContainerDied","Data":"c5b997edad21c5b50a86a7791394c810f2c565475608c7d94e24b11c0072dd55"} Oct 30 22:14:12.503120 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.502983 5039 scope.go:122] "RemoveContainer" containerID="3f82f170f4d14ded5ca998ded9700b01474d0844a4a950914fe12d1946bb1c78" Oct 30 22:14:12.503120 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.502940 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk" Oct 30 22:14:12.548268 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.548242 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk"] Oct 30 22:14:12.558039 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.558017 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ed0866-6bd7c66984-mcmgk"] Oct 30 22:14:12.703530 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.703505 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.921113 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.921091 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6ff71c-06fb-4546-a65e-d0ac053f8637" path="/var/lib/kubelet/pods/aa6ff71c-06fb-4546-a65e-d0ac053f8637/volumes" Oct 30 22:14:12.921559 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:12.921542 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" path="/var/lib/kubelet/pods/bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6/volumes" Oct 30 22:14:14.005548 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:14.005511 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:15.576498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:15.576465 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:15.813573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:15.813536 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:15.986631 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:15.986596 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:16.923156 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:16.923122 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524"] Oct 30 22:14:16.926258 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:16.924638 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" podUID="c714c6f5-861b-4659-b9fa-d4dcef2f58c3" containerName="maas-api" containerID="cri-o://219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0" gracePeriod=30 Oct 30 22:14:17.061298 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.061268 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.208246 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.208220 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 22:14:17.361065 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.361033 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs\") pod \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " Oct 30 22:14:17.361228 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.361087 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls\") pod \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " Oct 30 22:14:17.361228 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.361175 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca\") pod \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " Oct 30 22:14:17.361350 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.361288 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv\" (UniqueName: \"kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv\") pod \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\" (UID: \"c714c6f5-861b-4659-b9fa-d4dcef2f58c3\") " Oct 30 22:14:17.361560 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.361535 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca" pod "c714c6f5-861b-4659-b9fa-d4dcef2f58c3" (UID: "c714c6f5-861b-4659-b9fa-d4dcef2f58c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:17.363160 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.363137 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls" pod "c714c6f5-861b-4659-b9fa-d4dcef2f58c3" (UID: "c714c6f5-861b-4659-b9fa-d4dcef2f58c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:17.363264 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.363160 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs" pod "c714c6f5-861b-4659-b9fa-d4dcef2f58c3" (UID: "c714c6f5-861b-4659-b9fa-d4dcef2f58c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:17.363264 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.363201 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv" pod "c714c6f5-861b-4659-b9fa-d4dcef2f58c3" (UID: "c714c6f5-861b-4659-b9fa-d4dcef2f58c3"). InnerVolumeSpecName "kube-api-access-pldzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:17.463049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.462953 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pldzv\" (UniqueName: \"kubernetes.io/projected/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-kube-api-access-pldzv\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:17.463049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.462990 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:17.463049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.463003 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:17.463049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.463017 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c714c6f5-861b-4659-b9fa-d4dcef2f58c3-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:17.533506 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.533467 5039 generic.go:356] "Generic (PLEG): container finished" podID="c714c6f5-861b-4659-b9fa-d4dcef2f58c3" containerID="219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0" exitCode=0 Oct 30 22:14:17.533681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.533521 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" Oct 30 22:14:17.533681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.533557 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" event={"ID":"c714c6f5-861b-4659-b9fa-d4dcef2f58c3","Type":"ContainerDied","Data":"219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0"} Oct 30 22:14:17.533681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.533630 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524" event={"ID":"c714c6f5-861b-4659-b9fa-d4dcef2f58c3","Type":"ContainerDied","Data":"18d1a0eac6a9086083b3d4946603050de96a8807cc1056651323db810b8a5ce0"} Oct 30 22:14:17.533681 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.533663 5039 scope.go:122] "RemoveContainer" containerID="219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0" Oct 30 22:14:17.544989 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.544948 5039 scope.go:122] "RemoveContainer" containerID="219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0" Oct 30 22:14:17.545303 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:17.545283 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0\": container with ID starting with 219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0 not found: ID does not exist" containerID="219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0" Oct 30 22:14:17.545435 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.545310 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0"} err="failed to get container status \"219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0\": rpc error: code = NotFound desc = could not find container \"219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0\": container with ID starting with 219a6713aff72c68cdcc808180744b089895d132efaf7a6acef6ed0813a4f2b0 not found: ID does not exist" Oct 30 22:14:17.578300 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.578277 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524"] Oct 30 22:14:17.587768 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.587741 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-bfabe1-56d57bf959-dl524"] Oct 30 22:14:17.618783 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.618757 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.793534 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:17.793454 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:18.198137 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:18.198099 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:18.356472 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:18.356441 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:18.509937 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:18.509872 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:14:18.922286 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:18.922255 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c714c6f5-861b-4659-b9fa-d4dcef2f58c3" path="/var/lib/kubelet/pods/c714c6f5-861b-4659-b9fa-d4dcef2f58c3/volumes" Oct 30 22:14:19.463577 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:19.463544 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:19.663902 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:19.663868 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:19.832326 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:19.832237 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.066929 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:20.066893 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.233450 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:20.233418 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.436471 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:20.436424 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.432982 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:21.432948 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.560831 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:21.560804 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:14:21.567768 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:21.567742 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:14:23.105784 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:23.105753 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:24.704744 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:24.704716 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:24.982058 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:24.981974 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:25.142896 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:25.142857 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:25.183693 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:25.183663 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:14:25.503783 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:25.503751 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:26.882613 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:26.882570 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:28.719879 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:28.719836 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:30.402567 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:30.402528 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b"] Oct 30 22:14:30.413037 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:30.413012 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw"] Oct 30 22:14:30.433728 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:30.433701 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v"] Oct 30 22:14:30.567801 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:30.567774 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:30.728521 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:30.728439 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.034705 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:31.034631 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:14:31.035756 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:31.035725 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.227823 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:31.227784 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.840453 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:31.840384 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:32.013561 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:32.013524 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:32.338483 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:32.338442 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:33.782626 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:33.782585 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.281418 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:34.281348 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.434363 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:34.434313 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.880634 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:34.880601 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.350824 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:36.350791 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.526070 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:36.526037 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:38.085934 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:38.085893 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:39.699960 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:39.699919 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:40.935316 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:40.935282 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:41.104962 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:41.104923 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:41.284386 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:41.284296 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:41.443426 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:41.443377 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:42.908166 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:42.908129 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:43.501942 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:43.501908 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:43.685713 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:43.685673 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:43.841437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:43.841316 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:44.174516 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:44.174490 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:45.422333 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.422290 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" containerID="cri-o://757ffcf1766cdf5e17eedde29858efacc91b82399de87c63b68a1a0009a5133f" gracePeriod=45 Oct 30 22:14:45.432261 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.432227 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" containerID="cri-o://52a39de17e834fe8c17ddc9b2f0a53abcdce306114f30e63ff73b597d76a6391" gracePeriod=45 Oct 30 22:14:45.649844 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.649753 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" containerID="cri-o://fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6" gracePeriod=45 Oct 30 22:14:45.657677 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.655565 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:45.700446 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.700418 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" event={"ID":"a5210006-c87e-440e-9a2f-a5db81390620","Type":"ContainerDied","Data":"52a39de17e834fe8c17ddc9b2f0a53abcdce306114f30e63ff73b597d76a6391"} Oct 30 22:14:45.700555 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.700422 5039 generic.go:356] "Generic (PLEG): container finished" podID="a5210006-c87e-440e-9a2f-a5db81390620" containerID="52a39de17e834fe8c17ddc9b2f0a53abcdce306114f30e63ff73b597d76a6391" exitCode=0 Oct 30 22:14:45.702005 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.701981 5039 generic.go:356] "Generic (PLEG): container finished" podID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerID="757ffcf1766cdf5e17eedde29858efacc91b82399de87c63b68a1a0009a5133f" exitCode=0 Oct 30 22:14:45.702105 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.702049 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" event={"ID":"711dd07d-b60e-4a12-82a6-da528fb1d22b","Type":"ContainerDied","Data":"757ffcf1766cdf5e17eedde29858efacc91b82399de87c63b68a1a0009a5133f"} Oct 30 22:14:45.763356 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.763331 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 22:14:45.766673 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.766652 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 22:14:45.918774 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.918749 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 22:14:45.933256 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933204 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache\") pod \"711dd07d-b60e-4a12-82a6-da528fb1d22b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " Oct 30 22:14:45.933355 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933282 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home\") pod \"711dd07d-b60e-4a12-82a6-da528fb1d22b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " Oct 30 22:14:45.933355 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933329 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs\" (UniqueName: \"kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs\") pod \"711dd07d-b60e-4a12-82a6-da528fb1d22b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " Oct 30 22:14:45.933501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933385 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm\") pod \"711dd07d-b60e-4a12-82a6-da528fb1d22b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " Oct 30 22:14:45.933501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933434 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir\") pod \"711dd07d-b60e-4a12-82a6-da528fb1d22b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " Oct 30 22:14:45.933501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933464 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home\") pod \"a5210006-c87e-440e-9a2f-a5db81390620\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " Oct 30 22:14:45.933501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933480 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache" pod "711dd07d-b60e-4a12-82a6-da528fb1d22b" (UID: "711dd07d-b60e-4a12-82a6-da528fb1d22b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.933501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933491 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8\" (UniqueName: \"kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8\") pod \"a5210006-c87e-440e-9a2f-a5db81390620\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933525 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache\") pod \"a5210006-c87e-440e-9a2f-a5db81390620\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933552 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home" pod "711dd07d-b60e-4a12-82a6-da528fb1d22b" (UID: "711dd07d-b60e-4a12-82a6-da528fb1d22b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933583 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs\" (UniqueName: \"kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs\") pod \"711dd07d-b60e-4a12-82a6-da528fb1d22b\" (UID: \"711dd07d-b60e-4a12-82a6-da528fb1d22b\") " Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933615 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm\") pod \"a5210006-c87e-440e-9a2f-a5db81390620\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933643 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir\") pod \"a5210006-c87e-440e-9a2f-a5db81390620\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933714 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs\") pod \"a5210006-c87e-440e-9a2f-a5db81390620\" (UID: \"a5210006-c87e-440e-9a2f-a5db81390620\") " Oct 30 22:14:45.933760 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.933722 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir" pod "711dd07d-b60e-4a12-82a6-da528fb1d22b" (UID: "711dd07d-b60e-4a12-82a6-da528fb1d22b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.934183 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.934162 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:45.934246 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.934188 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:45.934246 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.934202 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:45.934357 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.934295 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home" pod "a5210006-c87e-440e-9a2f-a5db81390620" (UID: "a5210006-c87e-440e-9a2f-a5db81390620"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.934357 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.934343 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache" pod "a5210006-c87e-440e-9a2f-a5db81390620" (UID: "a5210006-c87e-440e-9a2f-a5db81390620"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.934493 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.934461 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir" pod "a5210006-c87e-440e-9a2f-a5db81390620" (UID: "a5210006-c87e-440e-9a2f-a5db81390620"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.935722 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.935699 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm" pod "711dd07d-b60e-4a12-82a6-da528fb1d22b" (UID: "711dd07d-b60e-4a12-82a6-da528fb1d22b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.935851 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.935788 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8" pod "a5210006-c87e-440e-9a2f-a5db81390620" (UID: "a5210006-c87e-440e-9a2f-a5db81390620"). InnerVolumeSpecName "kube-api-access-22wt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:45.935916 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.935895 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs" pod "711dd07d-b60e-4a12-82a6-da528fb1d22b" (UID: "711dd07d-b60e-4a12-82a6-da528fb1d22b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:45.936001 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.935981 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs" pod "711dd07d-b60e-4a12-82a6-da528fb1d22b" (UID: "711dd07d-b60e-4a12-82a6-da528fb1d22b"). InnerVolumeSpecName "kube-api-access-4cpvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:45.936211 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.936192 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm" pod "a5210006-c87e-440e-9a2f-a5db81390620" (UID: "a5210006-c87e-440e-9a2f-a5db81390620"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:45.936211 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:45.936191 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs" pod "a5210006-c87e-440e-9a2f-a5db81390620" (UID: "a5210006-c87e-440e-9a2f-a5db81390620"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:46.034648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034621 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs\") pod \"34a22871-a835-4685-8adc-ebe3e1960c6f\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " Oct 30 22:14:46.034826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034673 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir\") pod \"34a22871-a835-4685-8adc-ebe3e1960c6f\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " Oct 30 22:14:46.034826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034712 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache\") pod \"34a22871-a835-4685-8adc-ebe3e1960c6f\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " Oct 30 22:14:46.034826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034742 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm\") pod \"34a22871-a835-4685-8adc-ebe3e1960c6f\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " Oct 30 22:14:46.034826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034813 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt\" (UniqueName: \"kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt\") pod \"34a22871-a835-4685-8adc-ebe3e1960c6f\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " Oct 30 22:14:46.035063 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034843 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home\") pod \"34a22871-a835-4685-8adc-ebe3e1960c6f\" (UID: \"34a22871-a835-4685-8adc-ebe3e1960c6f\") " Oct 30 22:14:46.035063 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.034989 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache" pod "34a22871-a835-4685-8adc-ebe3e1960c6f" (UID: "34a22871-a835-4685-8adc-ebe3e1960c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:46.035063 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035029 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir" pod "34a22871-a835-4685-8adc-ebe3e1960c6f" (UID: "34a22871-a835-4685-8adc-ebe3e1960c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:46.035224 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035174 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home" pod "34a22871-a835-4685-8adc-ebe3e1960c6f" (UID: "34a22871-a835-4685-8adc-ebe3e1960c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:46.035311 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035294 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035315 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711dd07d-b60e-4a12-82a6-da528fb1d22b-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035328 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711dd07d-b60e-4a12-82a6-da528fb1d22b-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035339 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035351 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035369 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035364 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wt8\" (UniqueName: \"kubernetes.io/projected/a5210006-c87e-440e-9a2f-a5db81390620-kube-api-access-22wt8\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035376 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035404 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035418 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cpvs\" (UniqueName: \"kubernetes.io/projected/711dd07d-b60e-4a12-82a6-da528fb1d22b-kube-api-access-4cpvs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035431 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035443 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5210006-c87e-440e-9a2f-a5db81390620-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.035666 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.035456 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5210006-c87e-440e-9a2f-a5db81390620-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.036723 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.036701 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm" pod "34a22871-a835-4685-8adc-ebe3e1960c6f" (UID: "34a22871-a835-4685-8adc-ebe3e1960c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:46.036822 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.036761 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt" pod "34a22871-a835-4685-8adc-ebe3e1960c6f" (UID: "34a22871-a835-4685-8adc-ebe3e1960c6f"). InnerVolumeSpecName "kube-api-access-wthgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:46.036822 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.036792 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs" pod "34a22871-a835-4685-8adc-ebe3e1960c6f" (UID: "34a22871-a835-4685-8adc-ebe3e1960c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:46.136347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.136321 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wthgt\" (UniqueName: \"kubernetes.io/projected/34a22871-a835-4685-8adc-ebe3e1960c6f-kube-api-access-wthgt\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.136347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.136345 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a22871-a835-4685-8adc-ebe3e1960c6f-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.136531 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.136355 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a22871-a835-4685-8adc-ebe3e1960c6f-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:46.708222 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.708150 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" Oct 30 22:14:46.708222 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.708182 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" event={"ID":"a5210006-c87e-440e-9a2f-a5db81390620","Type":"ContainerDied","Data":"599a35c312aaa09dcf39feeb3d1d092d711c4259a460711b83084ed439cb24d8"} Oct 30 22:14:46.708738 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.708242 5039 scope.go:122] "RemoveContainer" containerID="52a39de17e834fe8c17ddc9b2f0a53abcdce306114f30e63ff73b597d76a6391" Oct 30 22:14:46.709700 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.709643 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" Oct 30 22:14:46.709700 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.709668 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b" event={"ID":"711dd07d-b60e-4a12-82a6-da528fb1d22b","Type":"ContainerDied","Data":"f6fab2e56c8837b2314954f6bfe2f434d90fdcbc43a2724068365db5a71e2324"} Oct 30 22:14:46.711083 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.711059 5039 generic.go:356] "Generic (PLEG): container finished" podID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerID="fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6" exitCode=0 Oct 30 22:14:46.711223 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.711111 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" Oct 30 22:14:46.711223 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.711131 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" event={"ID":"34a22871-a835-4685-8adc-ebe3e1960c6f","Type":"ContainerDied","Data":"fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6"} Oct 30 22:14:46.711223 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.711160 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v" event={"ID":"34a22871-a835-4685-8adc-ebe3e1960c6f","Type":"ContainerDied","Data":"c1bb9eeaac976c1de6567c9167e7e8ce533b50d0b0073a1f7f1e4f1b2593bf6f"} Oct 30 22:14:46.714261 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.714227 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:46.717484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.717461 5039 scope.go:122] "RemoveContainer" containerID="757ffcf1766cdf5e17eedde29858efacc91b82399de87c63b68a1a0009a5133f" Oct 30 22:14:46.726791 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.726774 5039 scope.go:122] "RemoveContainer" containerID="fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6" Oct 30 22:14:46.733994 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.733981 5039 scope.go:122] "RemoveContainer" containerID="fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6" Oct 30 22:14:46.734227 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:46.734212 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6\": container with ID starting with fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6 not found: ID does not exist" containerID="fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6" Oct 30 22:14:46.734282 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.734233 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6"} err="failed to get container status \"fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6\": rpc error: code = NotFound desc = could not find container \"fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6\": container with ID starting with fc28eac20ff5280957a3b448862187fe77925b9adb82bdf81a3efe0a6a4992d6 not found: ID does not exist" Oct 30 22:14:46.786473 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.786451 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b"] Oct 30 22:14:46.791942 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.791922 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-2-w1-b31518-kserve-cc5b5fb6c-2g68b"] Oct 30 22:14:46.820831 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.820805 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v"] Oct 30 22:14:46.825312 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.825290 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/simulator-w1-b31518-kserve-6b8584c848-f7w4v"] Oct 30 22:14:46.854884 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.854862 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw"] Oct 30 22:14:46.859156 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.859137 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/distinct-w1-b31518-kserve-fcc7674d4-z2wkw"] Oct 30 22:14:46.921116 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.921086 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" path="/var/lib/kubelet/pods/34a22871-a835-4685-8adc-ebe3e1960c6f/volumes" Oct 30 22:14:46.921615 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.921595 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" path="/var/lib/kubelet/pods/711dd07d-b60e-4a12-82a6-da528fb1d22b/volumes" Oct 30 22:14:46.921981 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:46.921968 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5210006-c87e-440e-9a2f-a5db81390620" path="/var/lib/kubelet/pods/a5210006-c87e-440e-9a2f-a5db81390620/volumes" Oct 30 22:14:47.317321 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:47.317287 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:49.025890 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:49.025850 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:50.007444 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:50.007406 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:14:50.744097 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:50.744061 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.485381 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:52.485346 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.654493 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:52.654460 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.824662 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:52.824570 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.926043 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:52.926004 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz"] Oct 30 22:14:52.926227 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:52.926203 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" containerID="cri-o://1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09" gracePeriod=30 Oct 30 22:14:53.019217 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:53.019187 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:54.552948 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:54.552908 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:55.183606 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:55.183569 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:14:56.406430 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:56.406315 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:57.736605 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:57.704613 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q"] Oct 30 22:14:57.736605 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:57.704838 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" podUID="46047522-2428-48c0-a1d7-adb65bd728e4" containerName="maas-api" containerID="cri-o://1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24" gracePeriod=30 Oct 30 22:14:57.818843 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:57.818805 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:58.053487 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.053464 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 22:14:58.158757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.158724 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls\") pod \"46047522-2428-48c0-a1d7-adb65bd728e4\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " Oct 30 22:14:58.158904 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.158763 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9\" (UniqueName: \"kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9\") pod \"46047522-2428-48c0-a1d7-adb65bd728e4\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " Oct 30 22:14:58.158904 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.158885 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca\") pod \"46047522-2428-48c0-a1d7-adb65bd728e4\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " Oct 30 22:14:58.159047 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.159028 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs\") pod \"46047522-2428-48c0-a1d7-adb65bd728e4\" (UID: \"46047522-2428-48c0-a1d7-adb65bd728e4\") " Oct 30 22:14:58.159287 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.159262 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca" pod "46047522-2428-48c0-a1d7-adb65bd728e4" (UID: "46047522-2428-48c0-a1d7-adb65bd728e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:58.159491 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.159473 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46047522-2428-48c0-a1d7-adb65bd728e4-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.161711 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.161686 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls" pod "46047522-2428-48c0-a1d7-adb65bd728e4" (UID: "46047522-2428-48c0-a1d7-adb65bd728e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:58.161842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.161771 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs" pod "46047522-2428-48c0-a1d7-adb65bd728e4" (UID: "46047522-2428-48c0-a1d7-adb65bd728e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:58.161842 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.161791 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9" pod "46047522-2428-48c0-a1d7-adb65bd728e4" (UID: "46047522-2428-48c0-a1d7-adb65bd728e4"). InnerVolumeSpecName "kube-api-access-ld9f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:58.190379 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.190357 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 22:14:58.260685 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260656 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.260812 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260719 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.260812 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260747 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.260812 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260796 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.260996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260838 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.260996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260903 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.260996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.260968 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.261159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261036 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.261159 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261069 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert\") pod \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\" (UID: \"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7\") " Oct 30 22:14:58.261268 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261237 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:58.261470 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261431 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:58.261584 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261510 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.261584 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261527 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9f9\" (UniqueName: \"kubernetes.io/projected/46047522-2428-48c0-a1d7-adb65bd728e4-kube-api-access-ld9f9\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.261584 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261537 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.261584 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.261551 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46047522-2428-48c0-a1d7-adb65bd728e4-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.262733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.262550 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:58.262733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.262611 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:58.262733 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.262658 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:58.263244 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.263213 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:58.263609 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.263588 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:58.263720 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.263683 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:58.263921 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.263893 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l" pod "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" (UID: "18b9b2fe-f726-4915-b4a5-6cc2466ad5f7"). InnerVolumeSpecName "kube-api-access-rfk9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:58.296892 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.296823 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:58.362202 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362183 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362202 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362201 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362212 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362221 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362230 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362237 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfk9l\" (UniqueName: \"kubernetes.io/projected/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-kube-api-access-rfk9l\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362246 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.362323 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.362254 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:14:58.452061 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.452038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:58.617754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.617652 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:58.784982 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.784950 5039 generic.go:356] "Generic (PLEG): container finished" podID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerID="1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09" exitCode=0 Oct 30 22:14:58.785354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.784999 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" Oct 30 22:14:58.785354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.785031 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" event={"ID":"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7","Type":"ContainerDied","Data":"1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09"} Oct 30 22:14:58.785354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.785089 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz" event={"ID":"18b9b2fe-f726-4915-b4a5-6cc2466ad5f7","Type":"ContainerDied","Data":"25ce952da94b416f3665ca7157bb9e3c70e863555ec32624eebb7c6764672fbe"} Oct 30 22:14:58.785354 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.785122 5039 scope.go:122] "RemoveContainer" containerID="1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09" Oct 30 22:14:58.786763 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.786739 5039 generic.go:356] "Generic (PLEG): container finished" podID="46047522-2428-48c0-a1d7-adb65bd728e4" containerID="1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24" exitCode=0 Oct 30 22:14:58.786885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.786781 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" Oct 30 22:14:58.786885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.786820 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" event={"ID":"46047522-2428-48c0-a1d7-adb65bd728e4","Type":"ContainerDied","Data":"1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24"} Oct 30 22:14:58.786885 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.786871 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q" event={"ID":"46047522-2428-48c0-a1d7-adb65bd728e4","Type":"ContainerDied","Data":"51048b21a18c4dc791576986668896ecc507cecc93ed2fc01ef1a2ddfac508ec"} Oct 30 22:14:58.795410 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.795373 5039 scope.go:122] "RemoveContainer" containerID="09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611" Oct 30 22:14:58.803773 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.803753 5039 scope.go:122] "RemoveContainer" containerID="1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09" Oct 30 22:14:58.804027 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:58.804010 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09\": container with ID starting with 1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09 not found: ID does not exist" containerID="1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09" Oct 30 22:14:58.804096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.804039 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09"} err="failed to get container status \"1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09\": rpc error: code = NotFound desc = could not find container \"1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09\": container with ID starting with 1d5cb60c5acfc8b7d0d429be931f11bccf4b34d5ef252eea267926680e548b09 not found: ID does not exist" Oct 30 22:14:58.804096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.804063 5039 scope.go:122] "RemoveContainer" containerID="09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611" Oct 30 22:14:58.804298 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:58.804281 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611\": container with ID starting with 09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611 not found: ID does not exist" containerID="09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611" Oct 30 22:14:58.804337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.804304 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611"} err="failed to get container status \"09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611\": rpc error: code = NotFound desc = could not find container \"09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611\": container with ID starting with 09ff303ad1473751a17aa8da5ec96987af5be8b351b0ed568d3bb9ebf72ff611 not found: ID does not exist" Oct 30 22:14:58.804337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.804319 5039 scope.go:122] "RemoveContainer" containerID="1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24" Oct 30 22:14:58.812049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.812033 5039 scope.go:122] "RemoveContainer" containerID="1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24" Oct 30 22:14:58.812298 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:14:58.812281 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24\": container with ID starting with 1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24 not found: ID does not exist" containerID="1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24" Oct 30 22:14:58.812344 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.812305 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24"} err="failed to get container status \"1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24\": rpc error: code = NotFound desc = could not find container \"1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24\": container with ID starting with 1a88ff00dfc25a3048d355f3d483153e7400bed96d7db5b14432635e19779a24 not found: ID does not exist" Oct 30 22:14:58.834819 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.834788 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz"] Oct 30 22:14:58.844896 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.844871 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e06b90-openshift-default-555df9b48c-8dqwz"] Oct 30 22:14:58.873024 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.872977 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q"] Oct 30 22:14:58.889982 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.889951 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ae6492-5d55fb4f48-bp98q"] Oct 30 22:14:58.898226 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.898191 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:58.920583 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.920543 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" path="/var/lib/kubelet/pods/18b9b2fe-f726-4915-b4a5-6cc2466ad5f7/volumes" Oct 30 22:14:58.921066 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:14:58.921044 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46047522-2428-48c0-a1d7-adb65bd728e4" path="/var/lib/kubelet/pods/46047522-2428-48c0-a1d7-adb65bd728e4/volumes" Oct 30 22:15:00.157707 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.157675 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8"] Oct 30 22:15:00.158144 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158129 5039 cpu_manager.go:414] "containerMap: removing container" podUID="fa1bb836-9911-4a78-9534-214c4916bf49" containerName="maas-api" Oct 30 22:15:00.158194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158145 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1bb836-9911-4a78-9534-214c4916bf49" containerName="maas-api" Oct 30 22:15:00.158194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158160 5039 cpu_manager.go:414] "containerMap: removing container" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" Oct 30 22:15:00.158194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158165 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" Oct 30 22:15:00.158194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158180 5039 cpu_manager.go:414] "containerMap: removing container" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" Oct 30 22:15:00.158194 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158187 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158198 5039 cpu_manager.go:414] "containerMap: removing container" podUID="46047522-2428-48c0-a1d7-adb65bd728e4" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158203 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46047522-2428-48c0-a1d7-adb65bd728e4" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158210 5039 cpu_manager.go:414] "containerMap: removing container" podUID="ea9761f2-8a71-4202-881a-13c7572defdd" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158215 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9761f2-8a71-4202-881a-13c7572defdd" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158222 5039 cpu_manager.go:414] "containerMap: removing container" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158227 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158239 5039 cpu_manager.go:414] "containerMap: removing container" podUID="c714c6f5-861b-4659-b9fa-d4dcef2f58c3" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158243 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c714c6f5-861b-4659-b9fa-d4dcef2f58c3" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158252 5039 cpu_manager.go:414] "containerMap: removing container" podUID="4d128504-5584-4533-9f19-1c00d1c10c76" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158259 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d128504-5584-4533-9f19-1c00d1c10c76" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158267 5039 cpu_manager.go:414] "containerMap: removing container" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158271 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158279 5039 cpu_manager.go:414] "containerMap: removing container" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158283 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158293 5039 cpu_manager.go:414] "containerMap: removing container" podUID="933a2d98-249d-4af7-a3f7-a73b8303410c" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158298 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933a2d98-249d-4af7-a3f7-a73b8303410c" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158305 5039 cpu_manager.go:414] "containerMap: removing container" podUID="94a805cf-6f02-4e24-9205-1f729fc615fa" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158310 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a805cf-6f02-4e24-9205-1f729fc615fa" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158316 5039 cpu_manager.go:414] "containerMap: removing container" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158321 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158328 5039 cpu_manager.go:414] "containerMap: removing container" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158336 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158345 5039 cpu_manager.go:414] "containerMap: removing container" podUID="cb181f48-bb52-4eff-8a23-d792758d7f53" containerName="maas-api" Oct 30 22:15:00.158342 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158350 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb181f48-bb52-4eff-8a23-d792758d7f53" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158357 5039 cpu_manager.go:414] "containerMap: removing container" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158361 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158367 5039 cpu_manager.go:414] "containerMap: removing container" podUID="bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158371 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158380 5039 cpu_manager.go:414] "containerMap: removing container" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158384 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158403 5039 cpu_manager.go:414] "containerMap: removing container" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158411 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158422 5039 cpu_manager.go:414] "containerMap: removing container" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158426 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158434 5039 cpu_manager.go:414] "containerMap: removing container" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158438 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158445 5039 cpu_manager.go:414] "containerMap: removing container" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158449 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158456 5039 cpu_manager.go:414] "containerMap: removing container" podUID="182ef68a-10a3-4119-a059-3883a088fe64" containerName="discovery" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158460 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182ef68a-10a3-4119-a059-3883a088fe64" containerName="discovery" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158466 5039 cpu_manager.go:414] "containerMap: removing container" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158470 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158476 5039 cpu_manager.go:414] "containerMap: removing container" podUID="dcb6bb80-25ca-4984-82af-293b8f254cd3" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158480 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb6bb80-25ca-4984-82af-293b8f254cd3" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158489 5039 cpu_manager.go:414] "containerMap: removing container" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158496 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158505 5039 cpu_manager.go:414] "containerMap: removing container" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158510 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158521 5039 cpu_manager.go:414] "containerMap: removing container" podUID="aa6ff71c-06fb-4546-a65e-d0ac053f8637" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158525 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6ff71c-06fb-4546-a65e-d0ac053f8637" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158532 5039 cpu_manager.go:414] "containerMap: removing container" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158536 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158620 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6ff71c-06fb-4546-a65e-d0ac053f8637" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158629 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158634 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1bb836-9911-4a78-9534-214c4916bf49" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158639 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cc53e2-1f21-48a7-b95a-4d0aab2f3d76" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158644 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a22871-a835-4685-8adc-ebe3e1960c6f" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158652 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="46047522-2428-48c0-a1d7-adb65bd728e4" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158662 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9761f2-8a71-4202-881a-13c7572defdd" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158667 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba35083-ab20-4a9d-a265-1238b03c7d07" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158673 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="c714c6f5-861b-4659-b9fa-d4dcef2f58c3" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158678 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d128504-5584-4533-9f19-1c00d1c10c76" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158683 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158690 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b9b2fe-f726-4915-b4a5-6cc2466ad5f7" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158696 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="933a2d98-249d-4af7-a3f7-a73b8303410c" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158701 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a805cf-6f02-4e24-9205-1f729fc615fa" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158706 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="93938b5f-db5a-4a74-b3a3-7c7fd42c8362" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158710 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="436b8ce8-13a5-4b79-bd40-868bfb93947b" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158715 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb181f48-bb52-4eff-8a23-d792758d7f53" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158720 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="711dd07d-b60e-4a12-82a6-da528fb1d22b" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158727 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfddd2d5-b5c9-46ff-915f-7ce09e2ab5d6" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158733 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158740 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158745 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b968a90-d6dd-481f-b845-8bc0463ffae5" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158750 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb133100-05d0-4614-b31a-6f9851985807" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158755 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff95999-64cd-49b1-aa4a-cf11bb3eb20c" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158760 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="182ef68a-10a3-4119-a059-3883a088fe64" containerName="discovery" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158765 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3a77f5-d1fa-4b1c-aaaf-80788fad9e3f" containerName="istio-proxy" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158770 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb6bb80-25ca-4984-82af-293b8f254cd3" containerName="maas-api" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158775 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5210006-c87e-440e-9a2f-a5db81390620" containerName="main" Oct 30 22:15:00.159029 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.158780 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a56bfa-062d-4b01-9778-8d9954e1d517" containerName="istio-proxy" Oct 30 22:15:00.163172 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.163153 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf"] Oct 30 22:15:00.163315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.163294 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.166801 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.166780 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.195546 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.195520 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf"] Oct 30 22:15:00.198664 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.198639 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8"] Oct 30 22:15:00.232107 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.232079 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:00.273113 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.273090 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\"" Oct 30 22:15:00.281541 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.281516 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.281647 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.281580 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlww\" (UniqueName: \"kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.281720 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.281696 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.281859 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.281840 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwmwh\" (UniqueName: \"kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.383595 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.383541 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwmwh\" (UniqueName: \"kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.383787 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.383637 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.383787 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.383691 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlww\" (UniqueName: \"kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.383787 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.383729 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.384662 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.384638 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.384968 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.384919 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.403670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.403639 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlww\" (UniqueName: \"kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.403996 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.403973 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwmwh\" (UniqueName: \"kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.478570 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.478490 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:00.489703 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.489674 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:00.809164 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.809091 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8"] Oct 30 22:15:00.813709 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:15:00.813683 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660cd801_8b0f_468e_9a61_7d5361873c5b.slice/crio-bb1dd4a3d578d51127abe2c98920b7997d9ef8e7a9b96b245ecb55a112d059cb WatchSource:0}: Error finding container bb1dd4a3d578d51127abe2c98920b7997d9ef8e7a9b96b245ecb55a112d059cb: Status 404 returned error can't find the container with id bb1dd4a3d578d51127abe2c98920b7997d9ef8e7a9b96b245ecb55a112d059cb Oct 30 22:15:00.822514 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.822485 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:00.830717 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:00.830695 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf"] Oct 30 22:15:00.832404 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:15:00.832365 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230364cb_ca2a_4e32_b4ca_e51b3a864714.slice/crio-0014ac15dbaa68ab03cd2bc07589ab6dcfca3cd9dc1c2b9b6be9b13060bd535d WatchSource:0}: Error finding container 0014ac15dbaa68ab03cd2bc07589ab6dcfca3cd9dc1c2b9b6be9b13060bd535d: Status 404 returned error can't find the container with id 0014ac15dbaa68ab03cd2bc07589ab6dcfca3cd9dc1c2b9b6be9b13060bd535d Oct 30 22:15:01.023205 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.023137 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:01.216065 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.216030 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:01.458087 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.458056 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:01.810829 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.810742 5039 generic.go:356] "Generic (PLEG): container finished" podID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerID="c31233611778687b2ac6271d8738df3e00563d9b6224dc3d8fe298d729c87541" exitCode=60 Oct 30 22:15:01.810994 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.810831 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" event={"ID":"660cd801-8b0f-468e-9a61-7d5361873c5b","Type":"ContainerDied","Data":"c31233611778687b2ac6271d8738df3e00563d9b6224dc3d8fe298d729c87541"} Oct 30 22:15:01.810994 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.810928 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" event={"ID":"660cd801-8b0f-468e-9a61-7d5361873c5b","Type":"ContainerStarted","Data":"bb1dd4a3d578d51127abe2c98920b7997d9ef8e7a9b96b245ecb55a112d059cb"} Oct 30 22:15:01.811079 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.811059 5039 scope.go:122] "RemoveContainer" containerID="c31233611778687b2ac6271d8738df3e00563d9b6224dc3d8fe298d729c87541" Oct 30 22:15:01.812480 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.812448 5039 generic.go:356] "Generic (PLEG): container finished" podID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerID="c79e9d7d28663d3056dc7aff0691692c6d37ae33b03fbeb8e704cf935bc5a32c" exitCode=60 Oct 30 22:15:01.812585 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.812488 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" event={"ID":"230364cb-ca2a-4e32-b4ca-e51b3a864714","Type":"ContainerDied","Data":"c79e9d7d28663d3056dc7aff0691692c6d37ae33b03fbeb8e704cf935bc5a32c"} Oct 30 22:15:01.812585 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.812518 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" event={"ID":"230364cb-ca2a-4e32-b4ca-e51b3a864714","Type":"ContainerStarted","Data":"0014ac15dbaa68ab03cd2bc07589ab6dcfca3cd9dc1c2b9b6be9b13060bd535d"} Oct 30 22:15:01.812701 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:01.812680 5039 scope.go:122] "RemoveContainer" containerID="c79e9d7d28663d3056dc7aff0691692c6d37ae33b03fbeb8e704cf935bc5a32c" Oct 30 22:15:02.822819 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.822785 5039 generic.go:356] "Generic (PLEG): container finished" podID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerID="3807673ba82148845a4f3d25d9eb0e28383fbfbe73a17c7b4ae37668514f7795" exitCode=60 Oct 30 22:15:02.823238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.822869 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" event={"ID":"660cd801-8b0f-468e-9a61-7d5361873c5b","Type":"ContainerDied","Data":"3807673ba82148845a4f3d25d9eb0e28383fbfbe73a17c7b4ae37668514f7795"} Oct 30 22:15:02.823238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.822930 5039 scope.go:122] "RemoveContainer" containerID="c31233611778687b2ac6271d8738df3e00563d9b6224dc3d8fe298d729c87541" Oct 30 22:15:02.823238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.823201 5039 scope.go:122] "RemoveContainer" containerID="3807673ba82148845a4f3d25d9eb0e28383fbfbe73a17c7b4ae37668514f7795" Oct 30 22:15:02.823582 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:02.823515 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8_odh-ai-gateway-infra(660cd801-8b0f-468e-9a61-7d5361873c5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" Oct 30 22:15:02.825009 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.824986 5039 generic.go:356] "Generic (PLEG): container finished" podID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerID="830a9e08d54546fd3ab9fae1cf2ee5eda3d4ff4253ba8624c775e75271b4a499" exitCode=60 Oct 30 22:15:02.825119 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.825082 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" event={"ID":"230364cb-ca2a-4e32-b4ca-e51b3a864714","Type":"ContainerDied","Data":"830a9e08d54546fd3ab9fae1cf2ee5eda3d4ff4253ba8624c775e75271b4a499"} Oct 30 22:15:02.825383 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.825367 5039 scope.go:122] "RemoveContainer" containerID="830a9e08d54546fd3ab9fae1cf2ee5eda3d4ff4253ba8624c775e75271b4a499" Oct 30 22:15:02.825703 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:02.825680 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf_odh-ai-gateway-infra(230364cb-ca2a-4e32-b4ca-e51b3a864714)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" Oct 30 22:15:02.828833 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.828807 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:02.834915 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:02.834900 5039 scope.go:122] "RemoveContainer" containerID="c79e9d7d28663d3056dc7aff0691692c6d37ae33b03fbeb8e704cf935bc5a32c" Oct 30 22:15:03.833263 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:03.833223 5039 scope.go:122] "RemoveContainer" containerID="3807673ba82148845a4f3d25d9eb0e28383fbfbe73a17c7b4ae37668514f7795" Oct 30 22:15:03.833746 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:03.833473 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8_odh-ai-gateway-infra(660cd801-8b0f-468e-9a61-7d5361873c5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" Oct 30 22:15:03.835171 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:03.835145 5039 scope.go:122] "RemoveContainer" containerID="830a9e08d54546fd3ab9fae1cf2ee5eda3d4ff4253ba8624c775e75271b4a499" Oct 30 22:15:03.835350 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:03.835329 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf_odh-ai-gateway-infra(230364cb-ca2a-4e32-b4ca-e51b3a864714)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" Oct 30 22:15:04.181671 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:04.181639 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:04.341171 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:04.341127 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:04.540798 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:04.540723 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:05.836379 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:05.836334 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.047633 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:06.047598 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.084980 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:06.084951 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh"] Oct 30 22:15:06.085218 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:06.085194 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" containerID="cri-o://40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2" gracePeriod=30 Oct 30 22:15:06.237015 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:06.235359 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.526135 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:06.526047 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.730921 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:06.730883 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.555837 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:07.554800 5039 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:15:07.555837 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:07.554856 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:15:07.680227 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:07.680169 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.852006 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:07.851922 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.935709 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:07.935675 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8"] Oct 30 22:15:08.042011 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.041976 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:08.092710 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.092677 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:08.156020 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.155986 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca\") pod \"660cd801-8b0f-468e-9a61-7d5361873c5b\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " Oct 30 22:15:08.156179 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.156111 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh\" (UniqueName: \"kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh\") pod \"660cd801-8b0f-468e-9a61-7d5361873c5b\" (UID: \"660cd801-8b0f-468e-9a61-7d5361873c5b\") " Oct 30 22:15:08.156399 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.156353 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca" pod "660cd801-8b0f-468e-9a61-7d5361873c5b" (UID: "660cd801-8b0f-468e-9a61-7d5361873c5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:08.156584 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.156561 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660cd801-8b0f-468e-9a61-7d5361873c5b-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:08.158120 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.158095 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh" pod "660cd801-8b0f-468e-9a61-7d5361873c5b" (UID: "660cd801-8b0f-468e-9a61-7d5361873c5b"). InnerVolumeSpecName "kube-api-access-fwmwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:08.257384 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.257355 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwmwh\" (UniqueName: \"kubernetes.io/projected/660cd801-8b0f-468e-9a61-7d5361873c5b-kube-api-access-fwmwh\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:08.495739 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.495651 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:08.651482 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.651444 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:08.821671 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.821589 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:08.857437 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.855173 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:15:08.873380 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.873354 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" Oct 30 22:15:08.873619 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.873352 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8" event={"ID":"660cd801-8b0f-468e-9a61-7d5361873c5b","Type":"ContainerDied","Data":"bb1dd4a3d578d51127abe2c98920b7997d9ef8e7a9b96b245ecb55a112d059cb"} Oct 30 22:15:08.873619 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.873489 5039 scope.go:122] "RemoveContainer" containerID="3807673ba82148845a4f3d25d9eb0e28383fbfbe73a17c7b4ae37668514f7795" Oct 30 22:15:08.934214 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.934192 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8"] Oct 30 22:15:08.940895 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:08.940863 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889975-h42q8"] Oct 30 22:15:09.158358 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:09.158325 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:10.738200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:10.738164 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:10.909947 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:10.909910 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:10.922370 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:10.922341 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" path="/var/lib/kubelet/pods/660cd801-8b0f-468e-9a61-7d5361873c5b/volumes" Oct 30 22:15:11.025835 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.025765 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm"] Oct 30 22:15:11.026083 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.026052 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" containerID="cri-o://1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89" gracePeriod=30 Oct 30 22:15:11.113173 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.113145 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:11.307856 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.307771 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:11.338246 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.338224 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 22:15:11.495337 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495309 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495366 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495422 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495495 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495474 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495663 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495503 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495663 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495535 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495663 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495614 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495836 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495679 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495836 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495740 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket\") pod \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\" (UID: \"6023cae4-55b6-4b8c-baf3-18f30fd21fe2\") " Oct 30 22:15:11.495836 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495619 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:11.495836 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495820 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:11.495836 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495800 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:11.496096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.495908 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:11.496096 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.496075 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:11.496212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.496089 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.496212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.496112 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.496212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.496125 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.496212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.496142 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.497605 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.497578 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "kube-api-access-4mqrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:11.497705 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.497665 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:11.497792 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.497775 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:11.497879 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.497800 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo" pod "6023cae4-55b6-4b8c-baf3-18f30fd21fe2" (UID: "6023cae4-55b6-4b8c-baf3-18f30fd21fe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:15:11.538916 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.538890 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6"] Oct 30 22:15:11.597068 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.597018 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mqrn\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-kube-api-access-4mqrn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.597068 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.597039 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.597068 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.597048 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.597068 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.597055 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.597068 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.597063 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6023cae4-55b6-4b8c-baf3-18f30fd21fe2-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:11.695216 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.695186 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:11.891372 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.891347 5039 generic.go:356] "Generic (PLEG): container finished" podID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerID="40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2" exitCode=0 Oct 30 22:15:11.891841 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.891422 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" Oct 30 22:15:11.891841 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.891430 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" event={"ID":"6023cae4-55b6-4b8c-baf3-18f30fd21fe2","Type":"ContainerDied","Data":"40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2"} Oct 30 22:15:11.891841 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.891484 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh" event={"ID":"6023cae4-55b6-4b8c-baf3-18f30fd21fe2","Type":"ContainerDied","Data":"fb00f04c5543a1762da58c90fd5bde7bcef89997a7b99b40a9b8e6162b8c2c45"} Oct 30 22:15:11.891841 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.891506 5039 scope.go:122] "RemoveContainer" containerID="40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2" Oct 30 22:15:11.902257 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.902241 5039 scope.go:122] "RemoveContainer" containerID="40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2" Oct 30 22:15:11.902523 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:11.902505 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2\": container with ID starting with 40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2 not found: ID does not exist" containerID="40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2" Oct 30 22:15:11.902575 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.902530 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2"} err="failed to get container status \"40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2\": rpc error: code = NotFound desc = could not find container \"40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2\": container with ID starting with 40fef0382cbfc989ea256a529d4e0a40f7cecf936ac44aa5fc87abbb47c7e6b2 not found: ID does not exist" Oct 30 22:15:11.935321 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.935301 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh"] Oct 30 22:15:11.945210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:11.945187 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-gpvdh"] Oct 30 22:15:12.410385 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:12.410339 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:12.920484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:12.920455 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" path="/var/lib/kubelet/pods/6023cae4-55b6-4b8c-baf3-18f30fd21fe2/volumes" Oct 30 22:15:14.018877 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:14.018829 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:15.648006 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:15.647975 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:15.807852 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:15.807796 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:15.914033 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:15.914000 5039 scope.go:122] "RemoveContainer" containerID="830a9e08d54546fd3ab9fae1cf2ee5eda3d4ff4253ba8624c775e75271b4a499" Oct 30 22:15:16.133295 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.133255 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:16.278439 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.278416 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 22:15:16.437445 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437359 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437445 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437428 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437457 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437482 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437515 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437549 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437608 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437627 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.437650 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437640 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:16.438025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437654 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs\") pod \"d416065e-0217-4c68-b041-cdb3383d55e7\" (UID: \"d416065e-0217-4c68-b041-cdb3383d55e7\") " Oct 30 22:15:16.438025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437770 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:16.438025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437963 5039 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.438025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437981 5039 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-data\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.438025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437982 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:16.438025 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.437996 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:16.438255 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.438211 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:16.439817 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.439779 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "kube-api-access-nskqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:16.439930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.439821 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:16.439930 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.439889 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:16.440045 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.439957 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo" pod "d416065e-0217-4c68-b041-cdb3383d55e7" (UID: "d416065e-0217-4c68-b041-cdb3383d55e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:15:16.538875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538848 5039 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d416065e-0217-4c68-b041-cdb3383d55e7-istiod-ca-cert\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.538875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538870 5039 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-istio-token\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.538875 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538879 5039 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d416065e-0217-4c68-b041-cdb3383d55e7-istio-podinfo\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.539049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538887 5039 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-workload-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.539049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538895 5039 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-istio-envoy\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.539049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538902 5039 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d416065e-0217-4c68-b041-cdb3383d55e7-credential-socket\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.539049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.538911 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nskqh\" (UniqueName: \"kubernetes.io/projected/d416065e-0217-4c68-b041-cdb3383d55e7-kube-api-access-nskqh\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:16.552244 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.552213 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:16.924254 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.924222 5039 generic.go:356] "Generic (PLEG): container finished" podID="d416065e-0217-4c68-b041-cdb3383d55e7" containerID="1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89" exitCode=0 Oct 30 22:15:16.924679 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.924322 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" Oct 30 22:15:16.924679 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.924540 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" event={"ID":"d416065e-0217-4c68-b041-cdb3383d55e7","Type":"ContainerDied","Data":"1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89"} Oct 30 22:15:16.924679 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.924589 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm" event={"ID":"d416065e-0217-4c68-b041-cdb3383d55e7","Type":"ContainerDied","Data":"892a03aa34658c8b6eb586b4b644dd5a2ce37a805654b33d1a9f4483b83b5553"} Oct 30 22:15:16.924679 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.924616 5039 scope.go:122] "RemoveContainer" containerID="1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89" Oct 30 22:15:16.926745 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.926722 5039 generic.go:356] "Generic (PLEG): container finished" podID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerID="87727aba86b58c4527ba4067be93f3ea826cfe1fcdbee97f29d89b6e4fb8f113" exitCode=60 Oct 30 22:15:16.926927 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.926743 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" event={"ID":"230364cb-ca2a-4e32-b4ca-e51b3a864714","Type":"ContainerDied","Data":"87727aba86b58c4527ba4067be93f3ea826cfe1fcdbee97f29d89b6e4fb8f113"} Oct 30 22:15:16.927130 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.927110 5039 scope.go:122] "RemoveContainer" containerID="87727aba86b58c4527ba4067be93f3ea826cfe1fcdbee97f29d89b6e4fb8f113" Oct 30 22:15:16.927354 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:16.927333 5039 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf_odh-ai-gateway-infra(230364cb-ca2a-4e32-b4ca-e51b3a864714)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" Oct 30 22:15:16.941312 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.941297 5039 scope.go:122] "RemoveContainer" containerID="1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89" Oct 30 22:15:16.941565 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:16.941546 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89\": container with ID starting with 1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89 not found: ID does not exist" containerID="1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89" Oct 30 22:15:16.941619 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.941575 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89"} err="failed to get container status \"1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89\": rpc error: code = NotFound desc = could not find container \"1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89\": container with ID starting with 1ef6e2a731f133dd0219356d7857d1cef32b41ce6a5246cd715e0df1468c1e89 not found: ID does not exist" Oct 30 22:15:16.941619 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:16.941590 5039 scope.go:122] "RemoveContainer" containerID="830a9e08d54546fd3ab9fae1cf2ee5eda3d4ff4253ba8624c775e75271b4a499" Oct 30 22:15:17.014065 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.014027 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm"] Oct 30 22:15:17.021015 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.020994 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-mbwnm"] Oct 30 22:15:17.321210 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.321130 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:17.373200 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373167 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh"] Oct 30 22:15:17.373740 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373721 5039 cpu_manager.go:414] "containerMap: removing container" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerName="cleanup" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373742 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerName="cleanup" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373755 5039 cpu_manager.go:414] "containerMap: removing container" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373761 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373779 5039 cpu_manager.go:414] "containerMap: removing container" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373784 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373793 5039 cpu_manager.go:414] "containerMap: removing container" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerName="cleanup" Oct 30 22:15:17.373810 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373797 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerName="cleanup" Oct 30 22:15:17.374030 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373860 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerName="cleanup" Oct 30 22:15:17.374030 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373869 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="6023cae4-55b6-4b8c-baf3-18f30fd21fe2" containerName="istio-proxy" Oct 30 22:15:17.374030 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.373877 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" containerName="istio-proxy" Oct 30 22:15:17.378225 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.378209 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.400937 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.400349 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:17.401351 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.401327 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh"] Oct 30 22:15:17.513065 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.513038 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:17.547576 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.547548 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.547676 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.547588 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.547676 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.547621 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m48\" (UniqueName: \"kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.649030 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.649008 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.649138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.649040 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.649138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.649081 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m48\" (UniqueName: \"kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.649670 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.649648 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.651302 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.651286 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.668991 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.668970 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m48\" (UniqueName: \"kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48\") pod \"maas-api-revoke-sub-b976a105-h68jh\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.690914 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.690894 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:17.954023 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.953922 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:17.983049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.983027 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh"] Oct 30 22:15:17.985597 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:15:17.985569 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61f18da_bae0_4a17_bf19_a4756450bcfa.slice/crio-fb75f15b159d2b93d6a2fed2a13a9774b7c481a58ffa7610f3fb8b1caa13478e WatchSource:0}: Error finding container fb75f15b159d2b93d6a2fed2a13a9774b7c481a58ffa7610f3fb8b1caa13478e: Status 404 returned error can't find the container with id fb75f15b159d2b93d6a2fed2a13a9774b7c481a58ffa7610f3fb8b1caa13478e Oct 30 22:15:17.991555 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:17.991528 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf"] Oct 30 22:15:18.136757 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.136733 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:18.256821 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.256741 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww\" (UniqueName: \"kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww\") pod \"230364cb-ca2a-4e32-b4ca-e51b3a864714\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " Oct 30 22:15:18.256980 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.256865 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca\") pod \"230364cb-ca2a-4e32-b4ca-e51b3a864714\" (UID: \"230364cb-ca2a-4e32-b4ca-e51b3a864714\") " Oct 30 22:15:18.257201 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.257179 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca" pod "230364cb-ca2a-4e32-b4ca-e51b3a864714" (UID: "230364cb-ca2a-4e32-b4ca-e51b3a864714"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.258944 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.258917 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww" pod "230364cb-ca2a-4e32-b4ca-e51b3a864714" (UID: "230364cb-ca2a-4e32-b4ca-e51b3a864714"). InnerVolumeSpecName "kube-api-access-pwlww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.335903 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.335879 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:18.358622 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.358601 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230364cb-ca2a-4e32-b4ca-e51b3a864714-maas-api-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:18.358622 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.358621 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwlww\" (UniqueName: \"kubernetes.io/projected/230364cb-ca2a-4e32-b4ca-e51b3a864714-kube-api-access-pwlww\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:18.763776 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.763748 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:18.920728 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.920701 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d416065e-0217-4c68-b041-cdb3383d55e7" path="/var/lib/kubelet/pods/d416065e-0217-4c68-b041-cdb3383d55e7/volumes" Oct 30 22:15:18.940314 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.940292 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" event={"ID":"c61f18da-bae0-4a17-bf19-a4756450bcfa","Type":"ContainerStarted","Data":"ceaa147debf50d9ca8bc30d29451dd1199f0ec5dc52f6bda916c4b2ee8ca6104"} Oct 30 22:15:18.940446 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.940339 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" event={"ID":"c61f18da-bae0-4a17-bf19-a4756450bcfa","Type":"ContainerStarted","Data":"fb75f15b159d2b93d6a2fed2a13a9774b7c481a58ffa7610f3fb8b1caa13478e"} Oct 30 22:15:18.941572 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.941556 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" Oct 30 22:15:18.941653 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.941581 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf" event={"ID":"230364cb-ca2a-4e32-b4ca-e51b3a864714","Type":"ContainerDied","Data":"0014ac15dbaa68ab03cd2bc07589ab6dcfca3cd9dc1c2b9b6be9b13060bd535d"} Oct 30 22:15:18.941653 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.941622 5039 scope.go:122] "RemoveContainer" containerID="87727aba86b58c4527ba4067be93f3ea826cfe1fcdbee97f29d89b6e4fb8f113" Oct 30 22:15:18.981922 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:18.981880 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" podStartSLOduration=1.981869175 podStartE2EDuration="1.981869175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:18.97937391 +0000 UTC m=+3958.189313550" watchObservedRunningTime="2026-10-30 22:15:18.981869175 +0000 UTC m=+3958.191808852" Oct 30 22:15:19.013034 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:19.013011 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf"] Oct 30 22:15:19.018720 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:19.018673 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889975-485kf"] Oct 30 22:15:19.705500 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:19.705459 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:19.948865 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:19.948833 5039 generic.go:356] "Generic (PLEG): container finished" podID="c61f18da-bae0-4a17-bf19-a4756450bcfa" containerID="ceaa147debf50d9ca8bc30d29451dd1199f0ec5dc52f6bda916c4b2ee8ca6104" exitCode=0 Oct 30 22:15:19.949049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:19.948909 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" event={"ID":"c61f18da-bae0-4a17-bf19-a4756450bcfa","Type":"ContainerDied","Data":"ceaa147debf50d9ca8bc30d29451dd1199f0ec5dc52f6bda916c4b2ee8ca6104"} Oct 30 22:15:20.475893 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:20.475866 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:15:20.516102 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:20.516069 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:20.920896 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:20.920860 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" path="/var/lib/kubelet/pods/230364cb-ca2a-4e32-b4ca-e51b3a864714/volumes" Oct 30 22:15:21.088330 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.088305 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:21.184560 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.184488 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca\") pod \"c61f18da-bae0-4a17-bf19-a4756450bcfa\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " Oct 30 22:15:21.184708 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.184605 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca\") pod \"c61f18da-bae0-4a17-bf19-a4756450bcfa\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " Oct 30 22:15:21.184708 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.184658 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48\" (UniqueName: \"kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48\") pod \"c61f18da-bae0-4a17-bf19-a4756450bcfa\" (UID: \"c61f18da-bae0-4a17-bf19-a4756450bcfa\") " Oct 30 22:15:21.184827 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.184796 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca" pod "c61f18da-bae0-4a17-bf19-a4756450bcfa" (UID: "c61f18da-bae0-4a17-bf19-a4756450bcfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.184952 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.184934 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-openshift-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:21.186512 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.186486 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca" pod "c61f18da-bae0-4a17-bf19-a4756450bcfa" (UID: "c61f18da-bae0-4a17-bf19-a4756450bcfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:21.186624 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.186562 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48" pod "c61f18da-bae0-4a17-bf19-a4756450bcfa" (UID: "c61f18da-bae0-4a17-bf19-a4756450bcfa"). InnerVolumeSpecName "kube-api-access-72m48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.286144 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.286121 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61f18da-bae0-4a17-bf19-a4756450bcfa-maas-api-xks-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:21.286144 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.286142 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72m48\" (UniqueName: \"kubernetes.io/projected/c61f18da-bae0-4a17-bf19-a4756450bcfa-kube-api-access-72m48\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:21.585438 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.585345 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q"] Oct 30 22:15:21.600012 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.599983 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq"] Oct 30 22:15:21.961534 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.961510 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" Oct 30 22:15:21.961701 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.961507 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b976a105-h68jh" event={"ID":"c61f18da-bae0-4a17-bf19-a4756450bcfa","Type":"ContainerDied","Data":"fb75f15b159d2b93d6a2fed2a13a9774b7c481a58ffa7610f3fb8b1caa13478e"} Oct 30 22:15:21.961701 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:21.961610 5039 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb75f15b159d2b93d6a2fed2a13a9774b7c481a58ffa7610f3fb8b1caa13478e" Oct 30 22:15:23.314261 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:23.314229 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:24.904622 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:24.904581 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:25.182797 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:25.182716 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:15:26.557673 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.557620 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" containerID="cri-o://b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334" gracePeriod=45 Oct 30 22:15:26.607160 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.607129 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:26.817053 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.807743 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 22:15:26.934098 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934070 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx\" (UniqueName: \"kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx\") pod \"845dd177-f965-4e53-9463-d35664bbc4a3\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " Oct 30 22:15:26.934257 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934142 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm\") pod \"845dd177-f965-4e53-9463-d35664bbc4a3\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " Oct 30 22:15:26.934257 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934177 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir\") pod \"845dd177-f965-4e53-9463-d35664bbc4a3\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " Oct 30 22:15:26.934257 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934199 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs\") pod \"845dd177-f965-4e53-9463-d35664bbc4a3\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " Oct 30 22:15:26.934257 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934240 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home\") pod \"845dd177-f965-4e53-9463-d35664bbc4a3\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " Oct 30 22:15:26.934484 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934317 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache\") pod \"845dd177-f965-4e53-9463-d35664bbc4a3\" (UID: \"845dd177-f965-4e53-9463-d35664bbc4a3\") " Oct 30 22:15:26.934686 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934611 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache" pod "845dd177-f965-4e53-9463-d35664bbc4a3" (UID: "845dd177-f965-4e53-9463-d35664bbc4a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.934826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934684 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home" pod "845dd177-f965-4e53-9463-d35664bbc4a3" (UID: "845dd177-f965-4e53-9463-d35664bbc4a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.934826 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.934751 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir" pod "845dd177-f965-4e53-9463-d35664bbc4a3" (UID: "845dd177-f965-4e53-9463-d35664bbc4a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.936680 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.936655 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm" pod "845dd177-f965-4e53-9463-d35664bbc4a3" (UID: "845dd177-f965-4e53-9463-d35664bbc4a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.936781 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.936678 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx" pod "845dd177-f965-4e53-9463-d35664bbc4a3" (UID: "845dd177-f965-4e53-9463-d35664bbc4a3"). InnerVolumeSpecName "kube-api-access-m67vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:26.936781 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.936723 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs" pod "845dd177-f965-4e53-9463-d35664bbc4a3" (UID: "845dd177-f965-4e53-9463-d35664bbc4a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:26.988580 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.988556 5039 generic.go:356] "Generic (PLEG): container finished" podID="845dd177-f965-4e53-9463-d35664bbc4a3" containerID="b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334" exitCode=0 Oct 30 22:15:26.988684 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.988599 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" Oct 30 22:15:26.988684 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.988608 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" event={"ID":"845dd177-f965-4e53-9463-d35664bbc4a3","Type":"ContainerDied","Data":"b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334"} Oct 30 22:15:26.988684 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.988637 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6" event={"ID":"845dd177-f965-4e53-9463-d35664bbc4a3","Type":"ContainerDied","Data":"e70365c7a76ed5edaf5dc3ec95fa5c4cee6a0c63103fa72ef0a703b77179ad56"} Oct 30 22:15:26.988684 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.988653 5039 scope.go:122] "RemoveContainer" containerID="b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334" Oct 30 22:15:26.997912 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.997893 5039 scope.go:122] "RemoveContainer" containerID="b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334" Oct 30 22:15:26.998173 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:26.998154 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334\": container with ID starting with b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334 not found: ID does not exist" containerID="b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334" Oct 30 22:15:26.998236 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:26.998182 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334"} err="failed to get container status \"b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334\": rpc error: code = NotFound desc = could not find container \"b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334\": container with ID starting with b3378263501bfb41313a56c8531812587c825caca4e4357fdbf52c30f07c6334 not found: ID does not exist" Oct 30 22:15:27.036154 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036120 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m67vx\" (UniqueName: \"kubernetes.io/projected/845dd177-f965-4e53-9463-d35664bbc4a3-kube-api-access-m67vx\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:27.036154 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036148 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:27.036315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036162 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:27.036315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036180 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845dd177-f965-4e53-9463-d35664bbc4a3-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:27.036315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036193 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:27.036315 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036206 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845dd177-f965-4e53-9463-d35664bbc4a3-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:27.036693 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.036675 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6"] Oct 30 22:15:27.040999 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.040980 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/simulator-w4-ea40ee-kserve-5c4f4fd8c-nb2d6"] Oct 30 22:15:27.218998 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.218978 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2"] Oct 30 22:15:27.427847 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.427810 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:27.880485 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:27.880444 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:28.213504 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:28.213467 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:28.699554 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:28.699524 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:28.920461 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:28.920431 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" path="/var/lib/kubelet/pods/845dd177-f965-4e53-9463-d35664bbc4a3/volumes" Oct 30 22:15:29.139773 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:29.139743 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:29.382665 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:29.382635 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:32.219787 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:32.219736 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerName="payload-processing" containerID="cri-o://29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476" gracePeriod=25 Oct 30 22:15:32.844591 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:32.844558 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:15:36.431545 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.431500 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:36.604428 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.604365 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" containerID="cri-o://4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2" gracePeriod=45 Oct 30 22:15:36.618220 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.618186 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" containerID="cri-o://9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b" gracePeriod=45 Oct 30 22:15:36.668339 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.668312 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:36.888498 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.888467 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:36.914585 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.914561 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 22:15:36.917677 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:36.917658 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.021853 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir\") pod \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.021900 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm\") pod \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.021945 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs\") pod \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.021970 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home\") pod \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022043 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs\") pod \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022103 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm\") pod \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022128 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir\") pod \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022159 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn\" (UniqueName: \"kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn\") pod \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022191 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home\") pod \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022220 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir" pod "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" (UID: "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022237 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache\") pod \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\" (UID: \"212834dc-cb28-4598-a82f-1d1b4b18d0c9\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022288 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache\") pod \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022328 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l\" (UniqueName: \"kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l\") pod \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\" (UID: \"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3\") " Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.022690 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.025074 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs" pod "212834dc-cb28-4598-a82f-1d1b4b18d0c9" (UID: "212834dc-cb28-4598-a82f-1d1b4b18d0c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.025811 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs" pod "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" (UID: "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.026065 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir" pod "212834dc-cb28-4598-a82f-1d1b4b18d0c9" (UID: "212834dc-cb28-4598-a82f-1d1b4b18d0c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.026140 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache" pod "212834dc-cb28-4598-a82f-1d1b4b18d0c9" (UID: "212834dc-cb28-4598-a82f-1d1b4b18d0c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.026241 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache" pod "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" (UID: "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.026258 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home" pod "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" (UID: "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.026550 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.026468 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home" pod "212834dc-cb28-4598-a82f-1d1b4b18d0c9" (UID: "212834dc-cb28-4598-a82f-1d1b4b18d0c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.031116 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.031086 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm" pod "212834dc-cb28-4598-a82f-1d1b4b18d0c9" (UID: "212834dc-cb28-4598-a82f-1d1b4b18d0c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.031262 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.031149 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm" pod "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" (UID: "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:37.031476 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.031452 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn" pod "212834dc-cb28-4598-a82f-1d1b4b18d0c9" (UID: "212834dc-cb28-4598-a82f-1d1b4b18d0c9"). InnerVolumeSpecName "kube-api-access-khfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:37.031780 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.031757 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l" pod "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" (UID: "9bf6be14-5c6a-4657-ae5d-013b45d9f1d3"). InnerVolumeSpecName "kube-api-access-w266l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:37.041449 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.041422 5039 generic.go:356] "Generic (PLEG): container finished" podID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerID="9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b" exitCode=0 Oct 30 22:15:37.041551 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.041478 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" Oct 30 22:15:37.041551 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.041500 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" event={"ID":"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3","Type":"ContainerDied","Data":"9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b"} Oct 30 22:15:37.041551 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.041540 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq" event={"ID":"9bf6be14-5c6a-4657-ae5d-013b45d9f1d3","Type":"ContainerDied","Data":"a362648469fb6cdb1f919f9bf17e4b78315e13ac72668a3d12e67828c2db092a"} Oct 30 22:15:37.041726 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.041560 5039 scope.go:122] "RemoveContainer" containerID="9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b" Oct 30 22:15:37.043129 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.043106 5039 generic.go:356] "Generic (PLEG): container finished" podID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerID="4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2" exitCode=0 Oct 30 22:15:37.043243 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.043146 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" event={"ID":"212834dc-cb28-4598-a82f-1d1b4b18d0c9","Type":"ContainerDied","Data":"4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2"} Oct 30 22:15:37.043243 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.043174 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" event={"ID":"212834dc-cb28-4598-a82f-1d1b4b18d0c9","Type":"ContainerDied","Data":"6c2d3820b88a939b161b59e94ec4e56dce04928e120594fc3fe0a142e78f41b1"} Oct 30 22:15:37.043243 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.043231 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" Oct 30 22:15:37.052485 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.052467 5039 scope.go:122] "RemoveContainer" containerID="9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b" Oct 30 22:15:37.052772 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:37.052743 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b\": container with ID starting with 9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b not found: ID does not exist" containerID="9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b" Oct 30 22:15:37.052873 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.052780 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b"} err="failed to get container status \"9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b\": rpc error: code = NotFound desc = could not find container \"9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b\": container with ID starting with 9429b061945ffbd3c13683e39b485ce230db55de454336e8336fed62fba7126b not found: ID does not exist" Oct 30 22:15:37.052873 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.052805 5039 scope.go:122] "RemoveContainer" containerID="4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2" Oct 30 22:15:37.064324 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.064214 5039 scope.go:122] "RemoveContainer" containerID="4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2" Oct 30 22:15:37.065146 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:37.065121 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2\": container with ID starting with 4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2 not found: ID does not exist" containerID="4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2" Oct 30 22:15:37.065238 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.065155 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2"} err="failed to get container status \"4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2\": rpc error: code = NotFound desc = could not find container \"4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2\": container with ID starting with 4878e5039f40ffd388f08b6436f22c7fe0045500a990e45ca734f069f38848b2 not found: ID does not exist" Oct 30 22:15:37.104710 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.104683 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq"] Oct 30 22:15:37.112515 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.112489 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/premium-w2-9d677d-kserve-6b74745659-ddrjq"] Oct 30 22:15:37.123373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123351 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w266l\" (UniqueName: \"kubernetes.io/projected/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-kube-api-access-w266l\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123373 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123373 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123382 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123405 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123413 5039 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-tls-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123422 5039 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-dshm\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123430 5039 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-tmp-dir\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123437 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khfhn\" (UniqueName: \"kubernetes.io/projected/212834dc-cb28-4598-a82f-1d1b4b18d0c9-kube-api-access-khfhn\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123445 5039 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-home\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123452 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212834dc-cb28-4598-a82f-1d1b4b18d0c9-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.123543 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.123459 5039 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3-model-cache\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:37.141897 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.141875 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q"] Oct 30 22:15:37.146517 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.146494 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q"] Oct 30 22:15:37.864843 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:37.864814 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-w2-9d677d-kserve-77cd49777d-mgd2q" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:38.921055 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:38.921019 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" path="/var/lib/kubelet/pods/212834dc-cb28-4598-a82f-1d1b4b18d0c9/volumes" Oct 30 22:15:38.921561 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:38.921463 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" path="/var/lib/kubelet/pods/9bf6be14-5c6a-4657-ae5d-013b45d9f1d3/volumes" Oct 30 22:15:39.432028 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:39.431963 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:40.182527 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.182499 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c"] Oct 30 22:15:40.182883 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.182717 5039 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" podUID="b36efbae-5b8f-4d22-a6fe-6b460d5c3688" containerName="maas-api" containerID="cri-o://afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92" gracePeriod=30 Oct 30 22:15:40.298031 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.297990 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:40.443532 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.443466 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 22:15:40.552761 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.552732 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls\") pod \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " Oct 30 22:15:40.552907 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.552766 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs\") pod \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " Oct 30 22:15:40.552907 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.552793 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z\" (UniqueName: \"kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z\") pod \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " Oct 30 22:15:40.552983 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.552917 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca\") pod \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\" (UID: \"b36efbae-5b8f-4d22-a6fe-6b460d5c3688\") " Oct 30 22:15:40.553313 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.553283 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca" pod "b36efbae-5b8f-4d22-a6fe-6b460d5c3688" (UID: "b36efbae-5b8f-4d22-a6fe-6b460d5c3688"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:40.554846 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.554817 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs" pod "b36efbae-5b8f-4d22-a6fe-6b460d5c3688" (UID: "b36efbae-5b8f-4d22-a6fe-6b460d5c3688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:40.554978 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.554862 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls" pod "b36efbae-5b8f-4d22-a6fe-6b460d5c3688" (UID: "b36efbae-5b8f-4d22-a6fe-6b460d5c3688"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:40.554978 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.554930 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z" pod "b36efbae-5b8f-4d22-a6fe-6b460d5c3688" (UID: "b36efbae-5b8f-4d22-a6fe-6b460d5c3688"). InnerVolumeSpecName "kube-api-access-92p7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:40.653725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.653692 5039 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-maas-api-tls\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:40.653725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.653719 5039 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-metrics-certs\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:40.653725 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.653729 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92p7z\" (UniqueName: \"kubernetes.io/projected/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-kube-api-access-92p7z\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:40.653952 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.653739 5039 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36efbae-5b8f-4d22-a6fe-6b460d5c3688-openshift-service-ca\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:40.765422 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.765343 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:40.934019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:40.933988 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:41.067347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.067272 5039 generic.go:356] "Generic (PLEG): container finished" podID="b36efbae-5b8f-4d22-a6fe-6b460d5c3688" containerID="afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92" exitCode=0 Oct 30 22:15:41.067347 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.067315 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" Oct 30 22:15:41.067610 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.067363 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" event={"ID":"b36efbae-5b8f-4d22-a6fe-6b460d5c3688","Type":"ContainerDied","Data":"afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92"} Oct 30 22:15:41.067610 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.067441 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c" event={"ID":"b36efbae-5b8f-4d22-a6fe-6b460d5c3688","Type":"ContainerDied","Data":"e7a3c79500073b6d8c1eec65ebdaa97aaa22928fa2ebc3110fffa41fef93002e"} Oct 30 22:15:41.067610 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.067474 5039 scope.go:122] "RemoveContainer" containerID="afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92" Oct 30 22:15:41.078296 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.078280 5039 scope.go:122] "RemoveContainer" containerID="afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92" Oct 30 22:15:41.078537 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:41.078522 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92\": container with ID starting with afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92 not found: ID does not exist" containerID="afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92" Oct 30 22:15:41.078594 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.078543 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92"} err="failed to get container status \"afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92\": rpc error: code = NotFound desc = could not find container \"afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92\": container with ID starting with afb5149c00f70c8fb5bce1d306d8be5b990acf581ebddd8e4f89176600316b92 not found: ID does not exist" Oct 30 22:15:41.112211 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.112189 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c"] Oct 30 22:15:41.121895 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.121876 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9d677d-69c66dfb44-lws4c"] Oct 30 22:15:41.370205 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:41.370145 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:42.920577 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:42.920540 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36efbae-5b8f-4d22-a6fe-6b460d5c3688" path="/var/lib/kubelet/pods/b36efbae-5b8f-4d22-a6fe-6b460d5c3688/volumes" Oct 30 22:15:44.534036 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:44.533995 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9ngr9"] Oct 30 22:15:47.684463 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:47.684417 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:47.859262 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:47.859220 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:48.045898 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:48.045802 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:49.320374 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:49.320342 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xj7qz_49251fad-bf0e-414b-8747-e2fb0f78f446/manager/0.log" Oct 30 22:15:49.423422 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:49.423375 5039 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:49.881673 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:49.881633 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:50.095181 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:50.095151 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:50.474605 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:50.474580 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9ngr9_d31c935f-6b9a-4ace-a486-16dcbb32e567/limitador/0.log" Oct 30 22:15:51.114326 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:51.114292 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:52.136844 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.136813 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:15:52.464327 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.464306 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 22:15:52.553103 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.553081 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume\") pod \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " Oct 30 22:15:52.553234 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.553174 5039 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98\" (UniqueName: \"kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98\") pod \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\" (UID: \"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2\") " Oct 30 22:15:52.553432 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.553411 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume" pod "2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" (UID: "2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:52.555071 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.555052 5039 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98" pod "2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" (UID: "2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2"). InnerVolumeSpecName "kube-api-access-vkh98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:52.654683 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.654655 5039 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkh98\" (UniqueName: \"kubernetes.io/projected/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-kube-api-access-vkh98\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:52.654683 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:52.654679 5039 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2-plugins-config-volume\") on node \"ip-10-0-133-149.ec2.internal\" DevicePath \"\"" Oct 30 22:15:53.129030 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.129001 5039 generic.go:356] "Generic (PLEG): container finished" podID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerID="29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476" exitCode=0 Oct 30 22:15:53.129140 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.129055 5039 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" Oct 30 22:15:53.129140 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.129079 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" event={"ID":"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2","Type":"ContainerDied","Data":"29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476"} Oct 30 22:15:53.129140 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.129130 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2" event={"ID":"2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2","Type":"ContainerDied","Data":"7e4e1a2e96819e9cfdd80371b8a5beea47d6beaa888c8da83f7eaf5de45f692d"} Oct 30 22:15:53.129259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.129153 5039 scope.go:122] "RemoveContainer" containerID="29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476" Oct 30 22:15:53.137648 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.137621 5039 scope.go:122] "RemoveContainer" containerID="29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476" Oct 30 22:15:53.137995 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:53.137915 5039 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476\": container with ID starting with 29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476 not found: ID does not exist" containerID="29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476" Oct 30 22:15:53.137995 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.137944 5039 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476"} err="failed to get container status \"29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476\": rpc error: code = NotFound desc = could not find container \"29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476\": container with ID starting with 29bc22ee857f2f6742a6078d7a3ab77164a9c8b116ecfa14e6859cbcddd9f476 not found: ID does not exist" Oct 30 22:15:53.176573 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.176550 5039 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2"] Oct 30 22:15:53.186425 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:53.186348 5039 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9d677d-6f5b9dff8f-gvfj2"] Oct 30 22:15:54.920079 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:54.920044 5039 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" path="/var/lib/kubelet/pods/2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2/volumes" Oct 30 22:15:55.183998 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:15:55.183928 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:15:56.837952 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:56.837916 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:58.254167 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:58.254132 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:58.446344 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:15:58.446309 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:00.259512 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:00.259476 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:05.281606 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:05.281527 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:05.996571 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:05.996532 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:08.658215 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:08.658186 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b976a105-h68jh_c61f18da-bae0-4a17-bf19-a4756450bcfa/revoke-keys/0.log" Oct 30 22:16:09.227755 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:09.227726 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-td45p_96dc590d-ce5f-473e-b427-53438f710fdb/postgres/0.log" Oct 30 22:16:10.741288 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:10.741255 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:10.995894 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:10.995805 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xj7qz_49251fad-bf0e-414b-8747-e2fb0f78f446/manager/0.log" Oct 30 22:16:11.005523 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:11.005481 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:11.977259 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:11.977231 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9ngr9_d31c935f-6b9a-4ace-a486-16dcbb32e567/limitador/0.log" Oct 30 22:16:12.934430 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:12.934367 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-ptnml_1ee76772-f17b-4258-85ea-8f9fc29bc20b/istio-proxy/0.log" Oct 30 22:16:14.192050 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:14.192019 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-m9s5t_b60c92be-c745-435e-b16b-6caf9000e769/istio-proxy/0.log" Oct 30 22:16:14.402539 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:14.402487 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zj6zn_adc2e906-5a26-491a-bc4a-898e3a323816/istio-proxy/0.log" Oct 30 22:16:14.610138 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:14.610065 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-rd599_687c04da-8703-43db-8d77-dd7db44099bf/istio-proxy/0.log" Oct 30 22:16:15.201403 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:15.201370 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b4b6cbf9-7xzk8_7c7fd700-057c-4ceb-88c4-01620fa6329d/router/0.log" Oct 30 22:16:18.451027 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:18.450990 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:25.184523 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:16:25.184477 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:16:28.396043 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.395982 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:28.433230 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433201 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xkksr/must-gather-m9p84"] Oct 30 22:16:28.433698 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433680 5039 cpu_manager.go:414] "containerMap: removing container" podUID="c61f18da-bae0-4a17-bf19-a4756450bcfa" containerName="revoke-keys" Oct 30 22:16:28.433698 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433701 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61f18da-bae0-4a17-bf19-a4756450bcfa" containerName="revoke-keys" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433720 5039 cpu_manager.go:414] "containerMap: removing container" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433725 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433735 5039 cpu_manager.go:414] "containerMap: removing container" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433739 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433747 5039 cpu_manager.go:414] "containerMap: removing container" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433752 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433758 5039 cpu_manager.go:414] "containerMap: removing container" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433763 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433770 5039 cpu_manager.go:414] "containerMap: removing container" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerName="payload-processing" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433775 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerName="payload-processing" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433781 5039 cpu_manager.go:414] "containerMap: removing container" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433786 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433795 5039 cpu_manager.go:414] "containerMap: removing container" podUID="b36efbae-5b8f-4d22-a6fe-6b460d5c3688" containerName="maas-api" Oct 30 22:16:28.433803 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433800 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36efbae-5b8f-4d22-a6fe-6b460d5c3688" containerName="maas-api" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433859 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433866 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="660cd801-8b0f-468e-9a61-7d5361873c5b" containerName="cleanup" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433874 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="212834dc-cb28-4598-a82f-1d1b4b18d0c9" containerName="main" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433880 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="845dd177-f965-4e53-9463-d35664bbc4a3" containerName="main" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433886 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3574dd-52cb-4ab7-85d6-ff35d2b20bc2" containerName="payload-processing" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433891 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf6be14-5c6a-4657-ae5d-013b45d9f1d3" containerName="main" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433897 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36efbae-5b8f-4d22-a6fe-6b460d5c3688" containerName="maas-api" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433902 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433912 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61f18da-bae0-4a17-bf19-a4756450bcfa" containerName="revoke-keys" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433986 5039 cpu_manager.go:414] "containerMap: removing container" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.433991 5039 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.434212 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.434039 5039 memory_manager.go:365] "RemoveStaleState removing state" podUID="230364cb-ca2a-4e32-b4ca-e51b3a864714" containerName="cleanup" Oct 30 22:16:28.437114 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.437098 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.455687 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.455663 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkksr/must-gather-m9p84"] Oct 30 22:16:28.457379 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.457359 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkksr\"/\"openshift-service-ca.crt\"" Oct 30 22:16:28.457501 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.457381 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkksr\"/\"kube-root-ca.crt\"" Oct 30 22:16:28.461245 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.461217 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xkksr\"/\"default-dockercfg-pmqwr\"" Oct 30 22:16:28.539361 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.539340 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6n5n\" (UniqueName: \"kubernetes.io/projected/4196923c-eb19-4bdb-8e35-b30e67ee8ba3-kube-api-access-c6n5n\") pod \"must-gather-m9p84\" (UID: \"4196923c-eb19-4bdb-8e35-b30e67ee8ba3\") " pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.539489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.539405 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4196923c-eb19-4bdb-8e35-b30e67ee8ba3-must-gather-output\") pod \"must-gather-m9p84\" (UID: \"4196923c-eb19-4bdb-8e35-b30e67ee8ba3\") " pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.556057 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.556031 5039 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:28.640216 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.640191 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6n5n\" (UniqueName: \"kubernetes.io/projected/4196923c-eb19-4bdb-8e35-b30e67ee8ba3-kube-api-access-c6n5n\") pod \"must-gather-m9p84\" (UID: \"4196923c-eb19-4bdb-8e35-b30e67ee8ba3\") " pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.640311 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.640243 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4196923c-eb19-4bdb-8e35-b30e67ee8ba3-must-gather-output\") pod \"must-gather-m9p84\" (UID: \"4196923c-eb19-4bdb-8e35-b30e67ee8ba3\") " pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.640564 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.640548 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4196923c-eb19-4bdb-8e35-b30e67ee8ba3-must-gather-output\") pod \"must-gather-m9p84\" (UID: \"4196923c-eb19-4bdb-8e35-b30e67ee8ba3\") " pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.657332 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.657270 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6n5n\" (UniqueName: \"kubernetes.io/projected/4196923c-eb19-4bdb-8e35-b30e67ee8ba3-kube-api-access-c6n5n\") pod \"must-gather-m9p84\" (UID: \"4196923c-eb19-4bdb-8e35-b30e67ee8ba3\") " pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:28.750566 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:28.750543 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkksr/must-gather-m9p84" Oct 30 22:16:29.044021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:29.043995 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkksr/must-gather-m9p84"] Oct 30 22:16:29.046411 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:16:29.046369 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4196923c_eb19_4bdb_8e35_b30e67ee8ba3.slice/crio-fcc206f29cbdcd90f15b853b5e15dff8aa0abaa08eb37a0cce1de4f255ffbca1 WatchSource:0}: Error finding container fcc206f29cbdcd90f15b853b5e15dff8aa0abaa08eb37a0cce1de4f255ffbca1: Status 404 returned error can't find the container with id fcc206f29cbdcd90f15b853b5e15dff8aa0abaa08eb37a0cce1de4f255ffbca1 Oct 30 22:16:29.048482 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:29.048464 5039 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:16:29.320783 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:29.320721 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkksr/must-gather-m9p84" event={"ID":"4196923c-eb19-4bdb-8e35-b30e67ee8ba3","Type":"ContainerStarted","Data":"fcc206f29cbdcd90f15b853b5e15dff8aa0abaa08eb37a0cce1de4f255ffbca1"} Oct 30 22:16:32.908417 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:32.908372 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b9l4r_4142b823-6b14-49f2-bf14-e8231f9296f3/global-pull-secret-syncer/0.log" Oct 30 22:16:33.228231 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:33.228165 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bnjl2_d9781bb5-7481-4803-8ef9-2eca6954c446/connection-checker/0.log" Oct 30 22:16:33.352019 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:33.351985 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j9ct9_c837b6b5-8ffc-4265-8d5c-1cc96e97fe75/connection-checker/0.log" Oct 30 22:16:33.462447 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:33.462419 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kmzkx_1ce09dc0-573a-4ac2-a6aa-6a37bead48fe/connection-checker/0.log" Oct 30 22:16:33.641854 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:33.641826 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hsqfc_2459def9-9723-4de1-9a93-5a2dfb7a3cca/konnectivity-agent/0.log" Oct 30 22:16:34.156494 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:34.156465 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-149.ec2.internal_524d7c023acdf7f01932ca99a8915366/haproxy/0.log" Oct 30 22:16:34.354367 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:34.354330 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkksr/must-gather-m9p84" event={"ID":"4196923c-eb19-4bdb-8e35-b30e67ee8ba3","Type":"ContainerStarted","Data":"55f38890f0050ed207babaa795da951a4eaf76b43bf0911a96ef2dacff896437"} Oct 30 22:16:34.354633 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:34.354609 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkksr/must-gather-m9p84" event={"ID":"4196923c-eb19-4bdb-8e35-b30e67ee8ba3","Type":"ContainerStarted","Data":"fcec93d3528b9d7ed3d7ce9ea399717723648f164d4b8a169e25b369cec2ecd0"} Oct 30 22:16:34.390623 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:34.390568 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xkksr/must-gather-m9p84" podStartSLOduration=1.958155654 podStartE2EDuration="6.390551346s" totalImagesPullingTime="4.432395692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:34.388520883 +0000 UTC m=+4033.598460512" watchObservedRunningTime="2026-10-30 22:16:34.390551346 +0000 UTC m=+4033.600490974" Oct 30 22:16:36.886929 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:36.886885 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:36.914744 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:36.914714 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:40.541469 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:40.541432 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xj7qz_49251fad-bf0e-414b-8747-e2fb0f78f446/manager/0.log" Oct 30 22:16:41.976066 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:41.976037 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9ngr9_d31c935f-6b9a-4ace-a486-16dcbb32e567/limitador/0.log" Oct 30 22:16:42.558386 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:42.558353 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:42.612846 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:42.612818 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:43.636575 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.636542 5039 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv"] Oct 30 22:16:43.645791 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.645765 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.661603 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.661576 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv"] Oct 30 22:16:43.664230 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.664205 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fx2nl\"/\"openshift-service-ca.crt\"" Oct 30 22:16:43.669245 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.669223 5039 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fx2nl\"/\"kube-root-ca.crt\"" Oct 30 22:16:43.669338 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.669293 5039 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fx2nl\"/\"default-dockercfg-sh698\"" Oct 30 22:16:43.772273 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.772145 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2k4z\" (UniqueName: \"kubernetes.io/projected/c38e20ee-cb51-4a69-b070-3464eec4b253-kube-api-access-z2k4z\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.772273 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.772203 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-podres\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.772273 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.772258 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-proc\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.772552 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.772299 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-sys\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.772552 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.772328 5039 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-lib-modules\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.873754 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873697 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2k4z\" (UniqueName: \"kubernetes.io/projected/c38e20ee-cb51-4a69-b070-3464eec4b253-kube-api-access-z2k4z\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.873886 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873788 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-podres\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.873886 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873848 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-proc\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.874021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873907 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-sys\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.874021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873917 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-podres\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.874021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873949 5039 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-lib-modules\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.874021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873989 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-proc\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.874021 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.873994 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-sys\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.874264 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.874096 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c38e20ee-cb51-4a69-b070-3464eec4b253-lib-modules\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.892444 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.892372 5039 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2k4z\" (UniqueName: \"kubernetes.io/projected/c38e20ee-cb51-4a69-b070-3464eec4b253-kube-api-access-z2k4z\") pod \"perf-node-gather-daemonset-p4wqv\" (UID: \"c38e20ee-cb51-4a69-b070-3464eec4b253\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:43.961130 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:43.961112 5039 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:44.277049 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:44.276906 5039 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv"] Oct 30 22:16:44.284456 ip-10-0-133-149 kubenswrapper[5039]: W1030 22:16:44.284424 5039 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc38e20ee_cb51_4a69_b070_3464eec4b253.slice/crio-23ba20a7a29aa30c25e8d8df4b904297e4c565bcec72945e7ad56836bebb0284 WatchSource:0}: Error finding container 23ba20a7a29aa30c25e8d8df4b904297e4c565bcec72945e7ad56836bebb0284: Status 404 returned error can't find the container with id 23ba20a7a29aa30c25e8d8df4b904297e4c565bcec72945e7ad56836bebb0284 Oct 30 22:16:44.420195 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:44.420152 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" event={"ID":"c38e20ee-cb51-4a69-b070-3464eec4b253","Type":"ContainerStarted","Data":"a3d19db666c135f99fddd38dfed90a5b8193100fdc1afdb9320ed062cdf6df27"} Oct 30 22:16:44.420377 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:44.420231 5039 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" event={"ID":"c38e20ee-cb51-4a69-b070-3464eec4b253","Type":"ContainerStarted","Data":"23ba20a7a29aa30c25e8d8df4b904297e4c565bcec72945e7ad56836bebb0284"} Oct 30 22:16:44.420377 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:44.420282 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:44.455701 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:44.455659 5039 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" podStartSLOduration=1.455647449 podStartE2EDuration="1.455647449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:44.454519547 +0000 UTC m=+4043.664459175" watchObservedRunningTime="2026-10-30 22:16:44.455647449 +0000 UTC m=+4043.665587083" Oct 30 22:16:45.180172 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:45.180142 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xr594_5abe8e87-10f7-453b-872f-ca366598dd92/cluster-monitoring-operator/0.log" Oct 30 22:16:46.598608 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:46.598576 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbmmk_a212b647-d67a-4fa5-a9a9-328d04378aea/node-exporter/0.log" Oct 30 22:16:46.726848 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:46.726816 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbmmk_a212b647-d67a-4fa5-a9a9-328d04378aea/kube-rbac-proxy/0.log" Oct 30 22:16:46.850553 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:46.850485 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbmmk_a212b647-d67a-4fa5-a9a9-328d04378aea/init-textfile/0.log" Oct 30 22:16:48.247772 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:48.247744 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:48.322628 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:48.322589 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-pg4ls_c112dc6f-ef7f-4aaf-9e71-38183285f0d8/manager/0.log" Oct 30 22:16:48.456118 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:48.456083 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-h46wg_b2586dac-e3ce-4d40-9ae1-52dacdd9acee/prometheus-operator-admission-webhook/0.log" Oct 30 22:16:50.437654 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:50.437582 5039 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-p4wqv" Oct 30 22:16:51.126131 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:51.126099 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-phllg_f71a2be6-1f45-43a6-ae97-8ebd007341fd/networking-console-plugin/0.log" Oct 30 22:16:52.131846 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:52.131802 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/1.log" Oct 30 22:16:52.140321 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:52.140295 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gh745_8fd6abf2-7709-4ef1-bed2-3f6b4b667bdf/console-operator/2.log" Oct 30 22:16:52.356967 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:52.356928 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-ptnml_1ee76772-f17b-4258-85ea-8f9fc29bc20b/istio-proxy/0.log" Oct 30 22:16:52.424356 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:52.424329 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-ptnml_1ee76772-f17b-4258-85ea-8f9fc29bc20b/istio-proxy/0.log" Oct 30 22:16:53.337084 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:53.337052 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-tsqlg_baac63f8-6718-46c9-a1de-2a382cf188c9/download-server/0.log" Oct 30 22:16:54.768741 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:54.768712 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-48pxc_2b8386a1-1b52-440d-b134-9c42390f70ba/volume-data-source-validator/0.log" Oct 30 22:16:55.184594 ip-10-0-133-149 kubenswrapper[5039]: E1030 22:16:55.184563 5039 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sq4d5" containerName="extractor" Oct 30 22:16:56.769240 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:56.769208 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-m9s5t_b60c92be-c745-435e-b16b-6caf9000e769/istio-proxy/0.log" Oct 30 22:16:56.832349 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:56.832323 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zj6zn_adc2e906-5a26-491a-bc4a-898e3a323816/istio-proxy/0.log" Oct 30 22:16:56.895995 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:56.895973 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-rd599_687c04da-8703-43db-8d77-dd7db44099bf/istio-proxy/0.log" Oct 30 22:16:57.097303 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:57.097236 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b4b6cbf9-7xzk8_7c7fd700-057c-4ceb-88c4-01620fa6329d/router/0.log" Oct 30 22:16:57.313404 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:57.313362 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rp8hx_7c5b111d-8074-4bda-8ac9-7d1f1f623c59/dns/0.log" Oct 30 22:16:57.425778 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:57.425748 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rp8hx_7c5b111d-8074-4bda-8ac9-7d1f1f623c59/kube-rbac-proxy/0.log" Oct 30 22:16:58.005489 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:58.005460 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-ptnml_1ee76772-f17b-4258-85ea-8f9fc29bc20b/istio-proxy/0.log" Oct 30 22:16:58.029182 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:58.029156 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-m9s5t_b60c92be-c745-435e-b16b-6caf9000e769/istio-proxy/0.log" Oct 30 22:16:58.096517 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:58.096481 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zj6zn_adc2e906-5a26-491a-bc4a-898e3a323816/istio-proxy/0.log" Oct 30 22:16:58.123052 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:58.123031 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c7kgv_9bafe23a-4bb7-477d-81a1-6024f5c3d9d4/dns-node-resolver/0.log" Oct 30 22:16:58.158752 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:58.158728 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-rd599_687c04da-8703-43db-8d77-dd7db44099bf/istio-proxy/0.log" Oct 30 22:16:58.360192 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:58.360120 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b4b6cbf9-7xzk8_7c7fd700-057c-4ceb-88c4-01620fa6329d/router/0.log" Oct 30 22:16:59.383697 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:59.383670 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f6cf8465-hcpb2_951cac7b-c383-4a73-b1cb-5fce1ff8a4f4/registry/0.log" Oct 30 22:16:59.502172 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:16:59.502140 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-87n87_22b8fafe-40b1-43de-b7f2-63a4f2cbfd0e/node-ca/0.log" Oct 30 22:17:01.278380 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.278329 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-m9s5t_b60c92be-c745-435e-b16b-6caf9000e769/istio-proxy/0.log" Oct 30 22:17:01.279322 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.279290 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xj7qz_49251fad-bf0e-414b-8747-e2fb0f78f446/manager/0.log" Oct 30 22:17:01.347692 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.341857 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zj6zn_adc2e906-5a26-491a-bc4a-898e3a323816/istio-proxy/0.log" Oct 30 22:17:01.398716 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.398688 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-rd599_687c04da-8703-43db-8d77-dd7db44099bf/istio-proxy/0.log" Oct 30 22:17:01.597469 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.597380 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9ngr9_d31c935f-6b9a-4ace-a486-16dcbb32e567/limitador/0.log" Oct 30 22:17:01.613838 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.613812 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b4b6cbf9-7xzk8_7c7fd700-057c-4ceb-88c4-01620fa6329d/router/0.log" Oct 30 22:17:01.626720 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:01.626699 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-ptnml_1ee76772-f17b-4258-85ea-8f9fc29bc20b/istio-proxy/0.log" Oct 30 22:17:02.171785 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:02.171756 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xj7qz_49251fad-bf0e-414b-8747-e2fb0f78f446/manager/0.log" Oct 30 22:17:02.602206 ip-10-0-133-149 kubenswrapper[5039]: I1030 22:17:02.602140 5039 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9ngr9_d31c935f-6b9a-4ace-a486-16dcbb32e567/limitador/0.log"